Max level shown:
12025-04-30T06:42:33.207ZDEBGtest_snapshot_basic (init_with_steps): Running step start_crdb
22025-04-30T06:42:33.208ZDEBGtest_snapshot_basic: Starting CRDB
32025-04-30T06:42:33.239ZINFOtest_snapshot_basic: cockroach temporary directory: /var/tmp/omicron_tmp/.tmpidNwvC
42025-04-30T06:42:33.239ZINFOtest_snapshot_basic: cockroach: copying from seed tarball (/var/tmp/omicron_tmp/crdb-base-build/f234518892194a05ad8767dc97f01a61a55baf1431942a314ce8b0f72b18ec2e.tar) to storage directory (/var/tmp/omicron_tmp/.tmpidNwvC/data)
52025-04-30T06:42:33.256ZINFOtest_snapshot_basic: cockroach command line: cockroach start-single-node --insecure --http-addr=:0 --store=path=/var/tmp/omicron_tmp/.tmpidNwvC/data,ballast-size=0 --listen-addr [::1]:0 --listening-url-file /var/tmp/omicron_tmp/.tmpidNwvC/listen-url
62025-04-30T06:42:33.256ZINFOtest_snapshot_basic: cockroach environment: BUILDOMAT_JOB_ID=01JT2JWKGHF3JHFMW0JR13NSA6 BUILDOMAT_TASK_ID=4 CARGO=/home/build/.rustup/toolchains/1.86.0-x86_64-unknown-illumos/bin/cargo CARGO_HOME=/home/build/.cargo CARGO_INCREMENTAL=0 CARGO_MANIFEST_DIR=/work/oxidecomputer/omicron/nexus CARGO_PKG_AUTHORS= CARGO_PKG_DESCRIPTION= CARGO_PKG_HOMEPAGE= CARGO_PKG_LICENSE=MPL-2.0 CARGO_PKG_LICENSE_FILE= CARGO_PKG_NAME=omicron-nexus CARGO_PKG_REPOSITORY= CARGO_PKG_RUST_VERSION= CARGO_PKG_VERSION=0.1.0 CARGO_PKG_VERSION_MAJOR=0 CARGO_PKG_VERSION_MINOR=1 CARGO_PKG_VERSION_PATCH=0 CARGO_PKG_VERSION_PRE= CARGO_TERM_COLOR=always CI=true CRDB_SEED_TAR=/var/tmp/omicron_tmp/crdb-base-build/f234518892194a05ad8767dc97f01a61a55baf1431942a314ce8b0f72b18ec2e.tar GITHUB_REPOSITORY=oxidecomputer/omicron GITHUB_SHA=531912b3fc900900bc7bd18f5eac1ce1e22cb1cd GOTRACEBACK=crash HOME=/home/build LANG=en_US.UTF-8 LC_ALL=en_US.UTF-8 LD_LIBRARY_PATH=/work/oxidecomputer/omicron/target/debug/build/aws-lc-sys-6532f52db341a7cc/out/build/artifacts:/work/oxidecomputer/omicron/target/debug/build/blake3-403ee89fc24db8f8/out:/work/oxidecomputer/omicron/target/debug/build/bzip2-sys-aece66bfb5c8df92/out/lib:/work/oxidecomputer/omicron/target/debug/build/libgit2-sys-5fe90ecb72e1690f/out/build:/work/oxidecomputer/omicron/target/debug/build/ring-fb368fe91b2bd145/out:/work/oxidecomputer/omicron/target/debug/build/tofino-ffe28f7f2301d0c2/out:/work/oxidecomputer/omicron/target/debug/build/zstd-sys-a7fb19a1e17b70f6/out:/work/oxidecomputer/omicron/target/debug/deps:/work/oxidecomputer/omicron/target/debug:/home/build/.rustup/toolchains/1.86.0-x86_64-unknown-illumos/lib/rustlib/x86_64-unknown-illumos/lib:/home/build/.rustup/toolchains/1.86.0-x86_64-unknown-illumos/lib LOGNAME=build NEXTEST=1 NEXTEST_BIN_EXE_nexus=/work/oxidecomputer/omicron/target/debug/nexus NEXTEST_BIN_EXE_schema-updater=/work/oxidecomputer/omicron/target/debug/schema-updater NEXTEST_EXECUTION_MODE=process-per-test NEXTEST_LD_LIBRARY_PATH=/work/oxidecomputer/omicron/target/debug/build/aws-lc-sys-6532f52db341a7cc/out/build/artifacts:/work/oxidecomputer/omicron/target/debug/build/blake3-403ee89fc24db8f8/out:/work/oxidecomputer/omicron/target/debug/build/bzip2-sys-aece66bfb5c8df92/out/lib:/work/oxidecomputer/omicron/target/debug/build/libgit2-sys-5fe90ecb72e1690f/out/build:/work/oxidecomputer/omicron/target/debug/build/ring-fb368fe91b2bd145/out:/work/oxidecomputer/omicron/target/debug/build/tofino-ffe28f7f2301d0c2/out:/work/oxidecomputer/omicron/target/debug/build/zstd-sys-a7fb19a1e17b70f6/out:/work/oxidecomputer/omicron/target/debug/deps:/work/oxidecomputer/omicron/target/debug:/home/build/.rustup/toolchains/1.86.0-x86_64-unknown-illumos/lib/rustlib/x86_64-unknown-illumos/lib:/home/build/.rustup/toolchains/1.86.0-x86_64-unknown-illumos/lib NEXTEST_PROFILE=ci NEXTEST_RUN_ID=29735cc2-0883-4059-8ca2-1da53f01ede3 NEXTEST_TEST_GLOBAL_SLOT=3 NEXTEST_TEST_GROUP=@global NEXTEST_TEST_GROUP_SLOT=none OUT_DIR=/work/oxidecomputer/omicron/target/debug/build/omicron-nexus-553062f859dbbfa0/out PATH=/work/oxidecomputer/omicron/out/mgd/root/opt/oxide/mgd/bin:/work/oxidecomputer/omicron/out/dendrite-stub/bin:/work/oxidecomputer/omicron/out/clickhouse:/work/oxidecomputer/omicron/out/cockroachdb/bin:/home/build/.cargo/bin:/usr/bin:/bin:/usr/sbin:/sbin:/opt/ooce/bin:/opt/ooce/sbin PWD=/work/oxidecomputer/omicron RUSTC_BOOTSTRAP=1 RUSTDOCFLAGS=--document-private-items -D warnings RUSTFLAGS=-D warnings -C link-arg=-R/usr/platform/oxide/lib/amd64 RUSTUP_HOME=/home/build/.rustup RUSTUP_TOOLCHAIN=1.86.0-x86_64-unknown-illumos RUST_BACKTRACE=1 RUST_RECURSION_COUNT=1 SHLVL=1 SSL_CERT_DIR=/usr/ssl/certs SSL_CERT_FILE=/etc/ssl/cacert.pem TMPDIR=/var/tmp/omicron_tmp TZ=UTC USER=build _=/usr/bin/ptime __NEXTEST_ATTEMPT=1
72025-04-30T06:42:34.930ZINFOtest_snapshot_basic: cockroach pid: 27015
82025-04-30T06:42:34.931ZINFOtest_snapshot_basic: cockroach listen URL: postgresql://root@[::1]:43851/omicron?sslmode=disable
92025-04-30T06:42:34.932ZDEBGtest_snapshot_basic (init_with_steps): Running step start_clickhouse
102025-04-30T06:42:34.932ZDEBGtest_snapshot_basic: Starting Clickhouse
112025-04-30T06:42:35.454ZDEBGtest_snapshot_basic (init_with_steps): Running step start_gateway_switch0
122025-04-30T06:42:35.454ZDEBGtest_snapshot_basic: Starting Management Gateway
132025-04-30T06:42:35.576ZDEBGtest_snapshot_basic (init_with_steps): Running step start_dendrite_switch0
142025-04-30T06:42:35.576ZDEBGtest_snapshot_basic: Starting Dendrite for switch0
152025-04-30T06:42:35.625ZDEBGtest_snapshot_basic (init_with_steps): Running step start_mgd_switch0
162025-04-30T06:42:35.625ZDEBGtest_snapshot_basic: Starting mgd for switch0
172025-04-30T06:42:35.737ZDEBGtest_snapshot_basic: mgd port is 41136
182025-04-30T06:42:35.737ZDEBGtest_snapshot_basic (init_with_steps): Running step record_switch_dns
192025-04-30T06:42:35.737ZDEBGtest_snapshot_basic: Recording DNS for the switch zones sled_id = b6d65341-167c-41df-9b5c-41cded99c229 switch_location = switch0
202025-04-30T06:42:35.738ZDEBGtest_snapshot_basic (init_with_steps): Running step start_internal_dns
212025-04-30T06:42:35.738ZINFOtest_snapshot_basic (store): opening sled database path = /var/tmp/omicron_tmp/.tmpZ1j6Qj
222025-04-30T06:42:35.842ZINFOtest_snapshot_basic (store): pruning trees for generations newer than 0
232025-04-30T06:42:35.842ZINFOtest_snapshot_basic (store): pruning trees for generations older than 0 keep = 3
242025-04-30T06:42:35.842ZINFOtest_snapshot_basic (dns): DNS server bound to address kind = dns local_address = [::1]:62208
252025-04-30T06:42:35.843ZDEBGtest_snapshot_basic (http): registered endpoint kind = dns local_addr = [::1]:48482 method = GET path = /config versions = all
262025-04-30T06:42:35.843ZDEBGtest_snapshot_basic (http): registered endpoint kind = dns local_addr = [::1]:48482 method = PUT path = /config versions = all
272025-04-30T06:42:35.843ZINFOtest_snapshot_basic (http): listening kind = dns local_addr = [::1]:48482
282025-04-30T06:42:35.844ZDEBGtest_snapshot_basic (http): successfully registered DTrace USDT probes kind = dns local_addr = [::1]:48482
292025-04-30T06:42:35.844ZDEBGtest_snapshot_basic (init_with_steps): Running step start_external_dns
302025-04-30T06:42:35.844ZINFOtest_snapshot_basic (store): opening sled database path = /var/tmp/omicron_tmp/.tmpF8qFRw
312025-04-30T06:42:35.857ZINFOtest_snapshot_basic (store): pruning trees for generations newer than 0
322025-04-30T06:42:35.857ZINFOtest_snapshot_basic (store): pruning trees for generations older than 0 keep = 3
332025-04-30T06:42:35.857ZINFOtest_snapshot_basic (dns): DNS server bound to address kind = dns local_address = [::1]:63945
342025-04-30T06:42:35.857ZDEBGtest_snapshot_basic (http): registered endpoint kind = dns local_addr = [::1]:46673 method = GET path = /config versions = all
352025-04-30T06:42:35.857ZDEBGtest_snapshot_basic (http): registered endpoint kind = dns local_addr = [::1]:46673 method = PUT path = /config versions = all
362025-04-30T06:42:35.857ZINFOtest_snapshot_basic (http): listening kind = dns local_addr = [::1]:46673
372025-04-30T06:42:35.858ZDEBGtest_snapshot_basic (http): successfully registered DTrace USDT probes kind = dns local_addr = [::1]:46673
382025-04-30T06:42:35.858ZDEBGtest_snapshot_basic (init_with_steps): Running step start_nexus_internal
392025-04-30T06:42:35.858ZDEBGtest_snapshot_basic: Starting Nexus (internal API)
402025-04-30T06:42:35.859ZINFOtest_snapshot_basic: setting up nexus server
412025-04-30T06:42:35.861ZINFOtest_snapshot_basic (ServerContext): registering Oso class class = Action
422025-04-30T06:42:35.861ZINFOtest_snapshot_basic (ServerContext): registering Oso class class = AnyActor
432025-04-30T06:42:35.861ZINFOtest_snapshot_basic (ServerContext): registering Oso class class = AuthenticatedActor
442025-04-30T06:42:35.861ZINFOtest_snapshot_basic (ServerContext): registering Oso class class = BlueprintConfig
452025-04-30T06:42:35.861ZINFOtest_snapshot_basic (ServerContext): registering Oso class class = Database
462025-04-30T06:42:35.861ZINFOtest_snapshot_basic (ServerContext): registering Oso class class = DnsConfig
472025-04-30T06:42:35.861ZINFOtest_snapshot_basic (ServerContext): registering Oso class class = Fleet
482025-04-30T06:42:35.862ZINFOtest_snapshot_basic (ServerContext): registering Oso class class = Inventory
492025-04-30T06:42:35.862ZINFOtest_snapshot_basic (ServerContext): registering Oso class class = IpPoolList
502025-04-30T06:42:35.862ZINFOtest_snapshot_basic (ServerContext): registering Oso class class = ConsoleSessionList
512025-04-30T06:42:35.862ZINFOtest_snapshot_basic (ServerContext): registering Oso class class = DeviceAuthRequestList
522025-04-30T06:42:35.862ZINFOtest_snapshot_basic (ServerContext): registering Oso class class = SiloCertificateList
532025-04-30T06:42:35.862ZINFOtest_snapshot_basic (ServerContext): registering Oso class class = SiloIdentityProviderList
542025-04-30T06:42:35.862ZINFOtest_snapshot_basic (ServerContext): registering Oso class class = SiloUserList
552025-04-30T06:42:35.862ZINFOtest_snapshot_basic (ServerContext): registering Oso class class = TargetReleaseConfig
562025-04-30T06:42:35.862ZINFOtest_snapshot_basic (ServerContext): registering Oso class class = WebhookEventClassList
572025-04-30T06:42:35.862ZINFOtest_snapshot_basic (ServerContext): registering Oso class class = Project
582025-04-30T06:42:35.862ZINFOtest_snapshot_basic (ServerContext): registering Oso class class = Disk
592025-04-30T06:42:35.862ZINFOtest_snapshot_basic (ServerContext): registering Oso class class = Snapshot
602025-04-30T06:42:35.862ZINFOtest_snapshot_basic (ServerContext): registering Oso class class = ProjectImage
612025-04-30T06:42:35.862ZINFOtest_snapshot_basic (ServerContext): registering Oso class class = AffinityGroup
622025-04-30T06:42:35.862ZINFOtest_snapshot_basic (ServerContext): registering Oso class class = AntiAffinityGroup
632025-04-30T06:42:35.862ZINFOtest_snapshot_basic (ServerContext): registering Oso class class = Instance
642025-04-30T06:42:35.862ZINFOtest_snapshot_basic (ServerContext): registering Oso class class = IpPool
652025-04-30T06:42:35.862ZINFOtest_snapshot_basic (ServerContext): registering Oso class class = InstanceNetworkInterface
662025-04-30T06:42:35.862ZINFOtest_snapshot_basic (ServerContext): registering Oso class class = Vpc
672025-04-30T06:42:35.862ZINFOtest_snapshot_basic (ServerContext): registering Oso class class = VpcRouter
682025-04-30T06:42:35.862ZINFOtest_snapshot_basic (ServerContext): registering Oso class class = InternetGateway
692025-04-30T06:42:35.862ZINFOtest_snapshot_basic (ServerContext): registering Oso class class = InternetGatewayIpPool
702025-04-30T06:42:35.862ZINFOtest_snapshot_basic (ServerContext): registering Oso class class = InternetGatewayIpAddress
712025-04-30T06:42:35.862ZINFOtest_snapshot_basic (ServerContext): registering Oso class class = RouterRoute
722025-04-30T06:42:35.862ZINFOtest_snapshot_basic (ServerContext): registering Oso class class = VpcSubnet
732025-04-30T06:42:35.862ZINFOtest_snapshot_basic (ServerContext): registering Oso class class = FloatingIp
742025-04-30T06:42:35.863ZINFOtest_snapshot_basic (ServerContext): registering Oso class class = Image
752025-04-30T06:42:35.863ZINFOtest_snapshot_basic (ServerContext): registering Oso class class = SiloImage
762025-04-30T06:42:35.863ZINFOtest_snapshot_basic (ServerContext): registering Oso class class = AddressLot
772025-04-30T06:42:35.863ZINFOtest_snapshot_basic (ServerContext): registering Oso class class = Blueprint
782025-04-30T06:42:35.863ZINFOtest_snapshot_basic (ServerContext): registering Oso class class = LoopbackAddress
792025-04-30T06:42:35.863ZINFOtest_snapshot_basic (ServerContext): registering Oso class class = Certificate
802025-04-30T06:42:35.863ZINFOtest_snapshot_basic (ServerContext): registering Oso class class = ConsoleSession
812025-04-30T06:42:35.863ZINFOtest_snapshot_basic (ServerContext): registering Oso class class = DeviceAuthRequest
822025-04-30T06:42:35.863ZINFOtest_snapshot_basic (ServerContext): registering Oso class class = DeviceAccessToken
832025-04-30T06:42:35.863ZINFOtest_snapshot_basic (ServerContext): registering Oso class class = PhysicalDisk
842025-04-30T06:42:35.863ZINFOtest_snapshot_basic (ServerContext): registering Oso class class = Rack
852025-04-30T06:42:35.863ZINFOtest_snapshot_basic (ServerContext): registering Oso class class = RoleBuiltin
862025-04-30T06:42:35.863ZINFOtest_snapshot_basic (ServerContext): registering Oso class class = SshKey
872025-04-30T06:42:35.863ZINFOtest_snapshot_basic (ServerContext): registering Oso class class = Silo
882025-04-30T06:42:35.863ZINFOtest_snapshot_basic (ServerContext): registering Oso class class = SiloUser
892025-04-30T06:42:35.863ZINFOtest_snapshot_basic (ServerContext): registering Oso class class = SiloGroup
902025-04-30T06:42:35.863ZINFOtest_snapshot_basic (ServerContext): registering Oso class class = SupportBundle
912025-04-30T06:42:35.863ZINFOtest_snapshot_basic (ServerContext): registering Oso class class = IdentityProvider
922025-04-30T06:42:35.863ZINFOtest_snapshot_basic (ServerContext): registering Oso class class = SamlIdentityProvider
932025-04-30T06:42:35.863ZINFOtest_snapshot_basic (ServerContext): registering Oso class class = Sled
942025-04-30T06:42:35.863ZINFOtest_snapshot_basic (ServerContext): registering Oso class class = TufRepo
952025-04-30T06:42:35.863ZINFOtest_snapshot_basic (ServerContext): registering Oso class class = TufArtifact
962025-04-30T06:42:35.863ZINFOtest_snapshot_basic (ServerContext): registering Oso class class = WebhookEvent
972025-04-30T06:42:35.863ZINFOtest_snapshot_basic (ServerContext): registering Oso class class = WebhookReceiver
982025-04-30T06:42:35.863ZINFOtest_snapshot_basic (ServerContext): registering Oso class class = WebhookSecret
992025-04-30T06:42:35.863ZINFOtest_snapshot_basic (ServerContext): registering Oso class class = Zpool
1002025-04-30T06:42:35.863ZINFOtest_snapshot_basic (ServerContext): registering Oso class class = Service
1012025-04-30T06:42:35.863ZINFOtest_snapshot_basic (ServerContext): registering Oso class class = UserBuiltin
1022025-04-30T06:42:35.863ZINFOtest_snapshot_basic (ServerContext): full Oso configuration config = #\n# Oso configuration for Omicron\n# This file is augmented by generated snippets.\n#\n\n#\n# ACTOR TYPES AND BASIC RULES\n#\n\n# `AnyActor` includes both authenticated and unauthenticated users.\nactor AnyActor {}\n\n# An `AuthenticatedActor` has an identity in the system. All of our operations\n# today require that an actor be authenticated.\nactor AuthenticatedActor {}\n\n# For any resource, `actor` can perform action `action` on it if they're\n# authenticated and their role(s) give them the corresponding permission on that\n# resource.\nallow(actor: AnyActor, action: Action, resource) if\n actor.authenticated and\n has_permission(actor.authn_actor.unwrap(), action.to_perm(), resource);\n\n# Define role relationships\nhas_role(actor: AuthenticatedActor, role: String, resource: Resource)\n\tif resource.has_role(actor, role);\n\n#\n# ROLES AND PERMISSIONS IN THE FLEET/SILO/PROJECT HIERARCHY\n#\n# We define the following permissions for most resources in the system:\n#\n# - "create_child": required to create child resources (of any type)\n#\n# - "list_children": required to list child resources (of all types) of a\n# resource\n#\n# - "modify": required to modify or delete a resource\n#\n# - "read": required to read a resource\n#\n# We define the following predefined roles for only a few high-level resources:\n# the Fleet (see below), Silo, Organization, and Project. The specific roles\n# are oriented around intended use-cases:\n#\n# - "admin": has all permissions on the resource\n#\n# - "collaborator": has "read", "list_children", and "create_child", plus\n# the "admin" role for child resources. The idea is that if you're an\n# Organization Collaborator, you have full control over the Projects within\n# the Organization, but you cannot modify or delete the Organization itself.\n#\n# - "viewer": has "read" and "list_children" on a resource\n#\n# Below the Project level, permissions are granted via roles at the Project\n# level. For example, for someone to be able to create, modify, or delete any\n# Instances, they must be granted project.collaborator, which means they can\n# create, modify, or delete _all_ resources in the Project.\n#\n# The complete set of predefined roles:\n#\n# - fleet.admin (superuser for the whole system)\n# - fleet.collaborator (can manage Silos)\n# - fleet.viewer (can read most non-siloed resources in the system)\n# - silo.admin (superuser for the silo)\n# - silo.collaborator (can create and own Organizations)\n# - silo.viewer (can read most resources within the Silo)\n# - organization.admin (complete control over an organization)\n# - organization.collaborator (can manage Projects)\n# - organization.viewer (can read most resources within the Organization)\n# - project.admin (complete control over a Project)\n# - project.collaborator (can manage all resources within the Project)\n# - project.viewer (can read most resources within the Project)\n#\n# Outside the Silo/Organization/Project hierarchy, we (currently) treat most\n# resources as nested under Fleet or else a synthetic resource (see below). We\n# do not yet support role assignments on anything other than Fleet, Silo,\n# Organization, or Project.\n#\n\n# "Fleet" is a global singleton representing the whole system. The name comes\n# from the idea described in RFD 24, but it's not quite right. This probably\n# should be more like "Region" or "AvailabilityZone". The precise boundaries\n# have not yet been figured out.\nresource Fleet {\n\tpermissions = [\n\t "list_children",\n\t "modify",\n\t "read",\n\t "create_child",\n\t];\n\n\troles = [\n\t # Roles that can be attached by users\n\t "admin",\n\t "collaborator",\n\t "viewer",\n\n\t # Internal-only roles\n\t "external-authenticator"\n\t];\n\n\t# Roles implied by other roles on this resource\n\t"viewer" if "collaborator";\n\t"collaborator" if "admin";\n\n\t# Permissions granted directly by roles on this resource\n\t"list_children" if "viewer";\n\t"read" if "viewer";\n\t"create_child" if "collaborator";\n\t"modify" if "admin";\n}\n\n# For fleets specifically, roles can be conferred by roles on the user's Silo.\nhas_role(actor: AuthenticatedActor, role: String, _: Fleet) if\n\tsilo_role in actor.confers_fleet_role(role) and\n\thas_role(actor, silo_role, actor.silo.unwrap());\n\nresource Silo {\n\tpermissions = [\n\t "list_children",\n\t "modify",\n\t "read",\n\t "create_child",\n\t];\n\troles = [ "admin", "collaborator", "viewer" ];\n\n\t# Roles implied by other roles on this resource\n\t"viewer" if "collaborator";\n\t"collaborator" if "admin";\n\n\t# Permissions granted directly by roles on this resource\n\t"list_children" if "viewer";\n\t"read" if "viewer";\n\n\t"create_child" if "collaborator";\n\t"modify" if "admin";\n\n\t# Permissions implied by roles on this resource's parent (Fleet). Fleet\n\t# privileges allow a user to see and potentially administer the Silo,\n\t# but they do not give anyone permission to look at anything inside the\n\t# Silo. To achieve this, we use permission rules here. (If we granted\n\t# Fleet administrators _roles_ on the Silo, then those would cascade\n\t# into the Silo as well.)\n\trelations = { parent_fleet: Fleet };\n\t"read" if "viewer" on "parent_fleet";\n\t"modify" if "collaborator" on "parent_fleet";\n\n\t# external authenticator has to create silo users\n\t"list_children" if "external-authenticator" on "parent_fleet";\n\t"create_child" if "external-authenticator" on "parent_fleet";\n}\n\nhas_relation(fleet: Fleet, "parent_fleet", silo: Silo)\n\tif silo.fleet = fleet;\n\n# As a special case, all authenticated users can read their own Silo. That's\n# not quite the same as having the "viewer" role. For example, they cannot list\n# Organizations in the Silo.\n#\n# One reason this is necessary is because if an unprivileged user tries to\n# create an Organization using "POST /organizations", they should get back a 403\n# (which implies they're able to see /organizations, which is essentially seeing\n# the Silo itself) rather than a 404. This behavior isn't a hard constraint\n# (i.e., you could reasonably get a 404 for an API you're not allowed to call).\n# Nor is the implementation (i.e., we could special-case this endpoint somehow).\n# But granting this permission is the simplest way to keep this endpoint's\n# behavior consistent with the rest of the API.\n#\n# This rule is also used to determine if a user can list the identity providers\n# in the Silo (which they should be able to), since that's predicated on being\n# able to read the Silo.\n#\n# It's unclear what else would break if users couldn't see their own Silo.\nhas_permission(actor: AuthenticatedActor, "read", silo: Silo)\n\tif silo in actor.silo;\n\nresource Project {\n\tpermissions = [\n\t "list_children",\n\t "modify",\n\t "read",\n\t "create_child",\n\t];\n\troles = [ "admin", "collaborator", "viewer" ];\n\n\t# Roles implied by other roles on this resource\n\t"viewer" if "collaborator";\n\t"collaborator" if "admin";\n\n\t# Permissions granted directly by roles on this resource\n\t"list_children" if "viewer";\n\t"read" if "viewer";\n\t"create_child" if "collaborator";\n\t"modify" if "admin";\n\n\t# Roles implied by roles on this resource's parent (Silo)\n\trelations = { parent_silo: Silo };\n\t"admin" if "collaborator" on "parent_silo";\n\t"viewer" if "viewer" on "parent_silo";\n}\nhas_relation(silo: Silo, "parent_silo", project: Project)\n\tif project.silo = silo;\n\n#\n# GENERAL RESOURCES OUTSIDE THE SILO/PROJECT HIERARCHY\n#\n# Many resources use snippets of Polar generated by the `authz_resource!` Rust\n# macro. Some resources require custom Polar code. Those appear here.\n#\n\nresource Certificate {\n\tpermissions = [ "read", "modify" ];\n\trelations = { parent_silo: Silo, parent_fleet: Fleet };\n\n\t# Fleet-level and silo-level roles both grant privileges on certificates.\n\t"read" if "admin" on "parent_silo";\n\t"modify" if "admin" on "parent_silo";\n\t"read" if "admin" on "parent_fleet";\n\t"modify" if "admin" on "parent_fleet";\n}\nhas_relation(silo: Silo, "parent_silo", certificate: Certificate)\n\tif certificate.silo = silo;\nhas_relation(fleet: Fleet, "parent_fleet", certificate: Certificate)\n\tif certificate.silo.fleet = fleet;\n\nresource SiloUser {\n\tpermissions = [\n\t "list_children",\n\t "modify",\n\t "read",\n\t "create_child",\n\t];\n\n\t# Fleet and Silo administrators can manage a Silo's users. This is one\n\t# of the only areas of Silo configuration that Fleet Administrators have\n\t# permissions on.\n\trelations = { parent_silo: Silo, parent_fleet: Fleet };\n\t"list_children" if "read" on "parent_silo";\n\t"read" if "read" on "parent_silo";\n\t"modify" if "admin" on "parent_silo";\n\t"create_child" if "admin" on "parent_silo";\n\t"list_children" if "read" on "parent_fleet";\n\t"read" if "read" on "parent_fleet";\n\t"modify" if "admin" on "parent_fleet";\n\t"create_child" if "admin" on "parent_fleet";\n}\nhas_relation(silo: Silo, "parent_silo", user: SiloUser)\n\tif user.silo = silo;\nhas_relation(fleet: Fleet, "parent_fleet", user: SiloUser)\n\tif user.silo.fleet = fleet;\n\n# authenticated actors have all permissions on themselves\nhas_permission(actor: AuthenticatedActor, _perm: String, silo_user: SiloUser)\n if actor.equals_silo_user(silo_user);\n\nhas_permission(actor: AuthenticatedActor, "read", silo_user: SiloUser)\n if silo_user.silo in actor.silo;\n\nresource SiloGroup {\n\tpermissions = [\n\t "list_children",\n\t "modify",\n\t "read",\n\t "create_child",\n\t];\n\n\trelations = { parent_silo: Silo };\n\t"list_children" if "read" on "parent_silo";\n\t"read" if "read" on "parent_silo";\n\t"modify" if "admin" on "parent_silo";\n\t"create_child" if "admin" on "parent_silo";\n}\nhas_relation(silo: Silo, "parent_silo", group: SiloGroup)\n\tif group.silo = silo;\n\nresource SshKey {\n\tpermissions = [ "read", "modify" ];\n\trelations = { silo_user: SiloUser };\n\n\t"read" if "read" on "silo_user";\n\t"modify" if "modify" on "silo_user";\n}\nhas_relation(user: SiloUser, "silo_user", ssh_key: SshKey)\n\tif ssh_key.silo_user = user;\n\nresource IdentityProvider {\n\tpermissions = [\n\t "read",\n\t "modify",\n\t "create_child",\n\t "list_children",\n\t];\n\trelations = { parent_silo: Silo, parent_fleet: Fleet };\n\n\t# Silo-level roles grant privileges on identity providers.\n\t"read" if "viewer" on "parent_silo";\n\t"list_children" if "viewer" on "parent_silo";\n\t"modify" if "admin" on "parent_silo";\n\t"create_child" if "admin" on "parent_silo";\n\n\t# Fleet-level roles also grant privileges on identity providers.\n\t"read" if "viewer" on "parent_fleet";\n\t"list_children" if "viewer" on "parent_fleet";\n\t"modify" if "admin" on "parent_fleet";\n\t"create_child" if "admin" on "parent_fleet";\n}\nhas_relation(silo: Silo, "parent_silo", identity_provider: IdentityProvider)\n\tif identity_provider.silo = silo;\nhas_relation(fleet: Fleet, "parent_fleet", collection: IdentityProvider)\n\tif collection.silo.fleet = fleet;\n\nresource SamlIdentityProvider {\n\tpermissions = [\n\t "read",\n\t "modify",\n\t "create_child",\n\t "list_children",\n\t];\n\trelations = { parent_silo: Silo, parent_fleet: Fleet };\n\n\t# Silo-level roles grant privileges on identity providers.\n\t"read" if "viewer" on "parent_silo";\n\t"list_children" if "viewer" on "parent_silo";\n\t"modify" if "admin" on "parent_silo";\n\t"create_child" if "admin" on "parent_silo";\n\n\t# Fleet-level roles also grant privileges on identity providers.\n\t"read" if "viewer" on "parent_fleet";\n\t"list_children" if "viewer" on "parent_fleet";\n\t"modify" if "admin" on "parent_fleet";\n\t"create_child" if "admin" on "parent_fleet";\n}\nhas_relation(silo: Silo, "parent_silo", saml_identity_provider: SamlIdentityProvider)\n\tif saml_identity_provider.silo = silo;\nhas_relation(fleet: Fleet, "parent_fleet", collection: SamlIdentityProvider)\n\tif collection.silo.fleet = fleet;\n\n#\n# SYNTHETIC RESOURCES OUTSIDE THE SILO HIERARCHY\n#\n# The resources here do not correspond to anything that appears explicitly in\n# the API or is stored in the database. These are used either at the top level\n# of the API path (e.g., "/v1/system/ip-pools") or as an implementation detail of the system\n# (in the case of console sessions and "Database"). The policies are\n# either statically-defined in this file or driven by role assignments on the\n# Fleet. None of these resources defines their own roles.\n#\n\n# Describes the policy for reading and modifying DNS configuration\n# (both internal and external)\nresource DnsConfig {\n\tpermissions = [ "read", "modify" ];\n\trelations = { parent_fleet: Fleet };\n\t# "external-authenticator" requires these permissions because that's the\n\t# context that Nexus uses when creating and deleting Silos. These\n\t# operations necessarily need to read and modify DNS configuration.\n\t"read" if "external-authenticator" on "parent_fleet";\n\t"modify" if "external-authenticator" on "parent_fleet";\n\t# "admin" on the parent fleet also gets these permissions, primarily for\n\t# the test suite.\n\t"read" if "admin" on "parent_fleet";\n\t"modify" if "admin" on "parent_fleet";\n}\nhas_relation(fleet: Fleet, "parent_fleet", dns_config: DnsConfig)\n\tif dns_config.fleet = fleet;\n\n# Describes the policy for accessing blueprints\nresource BlueprintConfig {\n\tpermissions = [\n\t "list_children", # list blueprints\n\t "create_child", # create blueprint\n\t "read", # read the current target\n\t "modify", # change the current target\n\t];\n\n\trelations = { parent_fleet: Fleet };\n\t"create_child" if "admin" on "parent_fleet";\n\t"modify" if "admin" on "parent_fleet";\n\t"list_children" if "viewer" on "parent_fleet";\n\t"read" if "viewer" on "parent_fleet";\n}\nhas_relation(fleet: Fleet, "parent_fleet", list: BlueprintConfig)\n\tif list.fleet = fleet;\n\n# Describes the policy for accessing blueprints\nresource TargetReleaseConfig {\n\tpermissions = [\n\t "read", # read the current target release\n\t "modify", # change the current target release\n\t];\n\n\trelations = { parent_fleet: Fleet };\n\t"read" if "viewer" on "parent_fleet";\n\t"modify" if "admin" on "parent_fleet";\n}\nhas_relation(fleet: Fleet, "parent_fleet", resource: TargetReleaseConfig)\n\tif resource.fleet = fleet;\n\n# Describes the policy for reading and modifying low-level inventory\nresource Inventory {\n\tpermissions = [ "read", "modify" ];\n\trelations = { parent_fleet: Fleet };\n\t"read" if "viewer" on "parent_fleet";\n\t"modify" if "admin" on "parent_fleet";\n}\nhas_relation(fleet: Fleet, "parent_fleet", inventory: Inventory)\n\tif inventory.fleet = fleet;\n\n# Describes the policy for accessing "/v1/system/ip-pools" in the API\nresource IpPoolList {\n\tpermissions = [\n\t "list_children",\n\t "modify",\n\t "create_child",\n\t];\n\n\t# Fleet Administrators can create or modify the IP Pools list.\n\trelations = { parent_fleet: Fleet };\n\t"modify" if "admin" on "parent_fleet";\n\t"create_child" if "admin" on "parent_fleet";\n\n\t# Fleet Viewers can list IP Pools\n\t"list_children" if "viewer" on "parent_fleet";\n}\nhas_relation(fleet: Fleet, "parent_fleet", ip_pool_list: IpPoolList)\n\tif ip_pool_list.fleet = fleet;\n\n# Any authenticated user can create a child of a provided IP Pool.\n# This is necessary to use the pools when provisioning instances.\nhas_permission(actor: AuthenticatedActor, "create_child", ip_pool: IpPool)\n\tif silo in actor.silo and silo.fleet = ip_pool.fleet;\n\n# Describes the policy for creating and managing web console sessions.\nresource ConsoleSessionList {\n\tpermissions = [ "create_child" ];\n\trelations = { parent_fleet: Fleet };\n\t"create_child" if "external-authenticator" on "parent_fleet";\n}\nhas_relation(fleet: Fleet, "parent_fleet", collection: ConsoleSessionList)\n\tif collection.fleet = fleet;\n\n# Describes the policy for creating and managing device authorization requests.\nresource DeviceAuthRequestList {\n\tpermissions = [ "create_child" ];\n\trelations = { parent_fleet: Fleet };\n\t"create_child" if "external-authenticator" on "parent_fleet";\n}\nhas_relation(fleet: Fleet, "parent_fleet", collection: DeviceAuthRequestList)\n\tif collection.fleet = fleet;\n\n# Describes the policy for creating and managing Silo certificates\nresource SiloCertificateList {\n\tpermissions = [ "list_children", "create_child" ];\n\n\trelations = { parent_silo: Silo, parent_fleet: Fleet };\n\n\t# Both Fleet and Silo administrators can see and modify the Silo's\n\t# certificates.\n\t"list_children" if "admin" on "parent_silo";\n\t"list_children" if "admin" on "parent_fleet";\n\t"create_child" if "admin" on "parent_silo";\n\t"create_child" if "admin" on "parent_fleet";\n}\nhas_relation(silo: Silo, "parent_silo", collection: SiloCertificateList)\n\tif collection.silo = silo;\nhas_relation(fleet: Fleet, "parent_fleet", collection: SiloCertificateList)\n\tif collection.silo.fleet = fleet;\n\n# Describes the policy for creating and managing Silo identity providers\nresource SiloIdentityProviderList {\n\tpermissions = [ "list_children", "create_child" ];\n\n\trelations = { parent_silo: Silo, parent_fleet: Fleet };\n\n\t# Everyone who can read the Silo (which includes all the users in the\n\t# Silo) can see the identity providers in it.\n\t"list_children" if "read" on "parent_silo";\n\n\t# Fleet and Silo administrators can manage the Silo's identity provider\n\t# configuration. This is one of the only areas of Silo configuration\n\t# that Fleet Administrators have permissions on. This is also one of\n\t# the only cases where we need to look two levels up the hierarchy to\n\t# see if somebody has the right permission. For most other things,\n\t# permissions cascade down the hierarchy so we only need to look at the\n\t# parent.\n\t"create_child" if "admin" on "parent_silo";\n\t"create_child" if "admin" on "parent_fleet";\n}\nhas_relation(silo: Silo, "parent_silo", collection: SiloIdentityProviderList)\n\tif collection.silo = silo;\nhas_relation(fleet: Fleet, "parent_fleet", collection: SiloIdentityProviderList)\n\tif collection.silo.fleet = fleet;\n\n# Describes the policy for creating and managing Silo users (mostly intended for\n# API-managed users)\nresource SiloUserList {\n\tpermissions = [ "list_children", "create_child" ];\n\n\trelations = { parent_silo: Silo, parent_fleet: Fleet };\n\n\t# Everyone who can read the Silo (which includes all the users in the\n\t# Silo) can see the users in it.\n\t"list_children" if "read" on "parent_silo";\n\n\t# Fleet and Silo administrators can manage the Silo's users. This is\n\t# one of the only areas of Silo configuration that Fleet Administrators\n\t# have permissions on. This is also one of the few cases (so far) where\n\t# we need to look two levels up the hierarchy to see if somebody has the\n\t# right permission. For most other things, permissions cascade down the\n\t# hierarchy so we only need to look at the parent.\n\t"create_child" if "admin" on "parent_silo";\n\t"list_children" if "admin" on "parent_fleet";\n\t"create_child" if "admin" on "parent_fleet";\n}\nhas_relation(silo: Silo, "parent_silo", collection: SiloUserList)\n\tif collection.silo = silo;\nhas_relation(fleet: Fleet, "parent_fleet", collection: SiloUserList)\n\tif collection.silo.fleet = fleet;\n\n# These rules grants the external authenticator role the permissions it needs to\n# read silo users and modify their sessions. This is necessary for login to\n# work.\nhas_permission(actor: AuthenticatedActor, "read", silo: Silo)\n\tif has_role(actor, "external-authenticator", silo.fleet);\nhas_permission(actor: AuthenticatedActor, "read", user: SiloUser)\n\tif has_role(actor, "external-authenticator", user.silo.fleet);\nhas_permission(actor: AuthenticatedActor, "modify", user: SiloUser)\n\tif has_role(actor, "external-authenticator", user.silo.fleet);\nhas_permission(actor: AuthenticatedActor, "read", group: SiloGroup)\n\tif has_role(actor, "external-authenticator", group.silo.fleet);\nhas_permission(actor: AuthenticatedActor, "modify", group: SiloGroup)\n\tif has_role(actor, "external-authenticator", group.silo.fleet);\n\nhas_permission(actor: AuthenticatedActor, "read", session: ConsoleSession)\n\tif has_role(actor, "external-authenticator", session.fleet);\nhas_permission(actor: AuthenticatedActor, "modify", session: ConsoleSession)\n\tif has_role(actor, "external-authenticator", session.fleet);\n\n# All authenticated users can read and delete device authn requests because\n# by necessity these operations happen before we've figured out what user (or\n# even Silo) the device auth is associated with. Any user can claim a device\n# auth request with the right user code (that's how it works) -- it's the user\n# code and associated logic that prevents unauthorized access here.\nhas_permission(_actor: AuthenticatedActor, "read", _device_auth: DeviceAuthRequest);\nhas_permission(_actor: AuthenticatedActor, "modify", _device_auth: DeviceAuthRequest);\n\nhas_permission(actor: AuthenticatedActor, "read", device_token: DeviceAccessToken)\n\tif has_role(actor, "external-authenticator", device_token.fleet);\n\nhas_permission(actor: AuthenticatedActor, "read", identity_provider: IdentityProvider)\n\tif has_role(actor, "external-authenticator", identity_provider.silo.fleet);\n\nhas_permission(actor: AuthenticatedActor, "read", saml_identity_provider: SamlIdentityProvider)\n\tif has_role(actor, "external-authenticator", saml_identity_provider.silo.fleet);\n\n# Describes the policy for who can access the internal database.\nresource Database {\n\tpermissions = [\n\t # "query" is required to perform any query against the database,\n\t # whether a read or write query. This is checked when an operation\n\t # checks out a database connection from the connection pool.\n\t #\n\t # Any authenticated user gets this permission. There's generally\n\t # some other authz check involved in the database query. For\n\t # example, if you're querying the database to "read" a "Project", we\n\t # should also be checking that. So why do we do this at all? It's\n\t # a belt-and-suspenders measure so that if we somehow introduced an\n\t # unauthenticated code path that hits the database, it cannot be\n\t # used to DoS the database because we won't allow the operation to\n\t # make the query. (As long as the code path _is_ authenticated, we\n\t # can use throttling mechanisms to prevent DoS.)\n\t "query",\n\n\t # "modify" is required to populate database data that's delivered\n\t # with the system. It should also be required for schema changes,\n\t # when we support those. This is separate from "query" so that we\n\t # cannot accidentally invoke these code paths from API calls and\n\t # other general functions.\n\t "modify"\n\t];\n}\n\n# All authenticated users have the "query" permission on the database.\nhas_permission(_actor: AuthenticatedActor, "query", _resource: Database);\n\n# The "db-init" user is the only one with the "modify" permission.\nhas_permission(USER_DB_INIT: AuthenticatedActor, "modify", _resource: Database);\nhas_permission(USER_DB_INIT: AuthenticatedActor, "create_child", _resource: IpPoolList);\n# It also has "admin" on the internal silo to populate it with built-in resources.\n# TODO-completeness: actually limit to just internal silo and not all silos\nhas_role(USER_DB_INIT: AuthenticatedActor, "admin", _silo: Silo);\n\n# Allow the internal API admin permissions on all silos.\nhas_role(USER_INTERNAL_API: AuthenticatedActor, "admin", _silo: Silo);\n\nresource WebhookSecret {\n\tpermissions = [ "read", "modify" ];\n\trelations = { parent_webhook_receiver: WebhookReceiver };\n\n\t"read" if "read" on "parent_webhook_receiver";\n\t"modify" if "modify" on "parent_webhook_receiver";\n}\n\nhas_relation(rx: WebhookReceiver, "parent_webhook_receiver", secret: WebhookSecret)\n\tif secret.webhook_receiver = rx;\n\nresource WebhookEventClassList {\n\tpermissions = [ "list_children" ];\n\trelations = { parent_fleet: Fleet };\n\n\t"list_children" if "viewer" on "parent_fleet";\n}\n\nhas_relation(fleet: Fleet, "parent_fleet", collection: WebhookEventClassList)\n\tif collection.fleet = fleet;\n\n\n\n resource Disk {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n\n relations = { containing_project: Project };\n "list_children" if "viewer" on "containing_project";\n "read" if "viewer" on "containing_project";\n "modify" if "collaborator" on "containing_project";\n "create_child" if "collaborator" on "containing_project";\n }\n\n has_relation(parent: Project, "containing_project", child: Disk)\n if child.project = parent;\n \n\n resource Snapshot {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n\n relations = { containing_project: Project };\n "list_children" if "viewer" on "containing_project";\n "read" if "viewer" on "containing_project";\n "modify" if "collaborator" on "containing_project";\n "create_child" if "collaborator" on "containing_project";\n }\n\n has_relation(parent: Project, "containing_project", child: Snapshot)\n if child.project = parent;\n \n\n resource ProjectImage {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n\n relations = { containing_project: Project };\n "list_children" if "viewer" on "containing_project";\n "read" if "viewer" on "containing_project";\n "modify" if "collaborator" on "containing_project";\n "create_child" if "collaborator" on "containing_project";\n }\n\n has_relation(parent: Project, "containing_project", child: ProjectImage)\n if child.project = parent;\n \n\n resource AffinityGroup {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n\n relations = { containing_project: Project };\n "list_children" if "viewer" on "containing_project";\n "read" if "viewer" on "containing_project";\n "modify" if "collaborator" on "containing_project";\n "create_child" if "collaborator" on "containing_project";\n }\n\n has_relation(parent: Project, "containing_project", child: AffinityGroup)\n if child.project = parent;\n \n\n resource AntiAffinityGroup {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n\n relations = { containing_project: Project };\n "list_children" if "viewer" on "containing_project";\n "read" if "viewer" on "containing_project";\n "modify" if "collaborator" on "containing_project";\n "create_child" if "collaborator" on "containing_project";\n }\n\n has_relation(parent: Project, "containing_project", child: AntiAffinityGroup)\n if child.project = parent;\n \n\n resource Instance {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n\n relations = { containing_project: Project };\n "list_children" if "viewer" on "containing_project";\n "read" if "viewer" on "containing_project";\n "modify" if "collaborator" on "containing_project";\n "create_child" if "collaborator" on "containing_project";\n }\n\n has_relation(parent: Project, "containing_project", child: Instance)\n if child.project = parent;\n \n\n resource IpPool {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n \n relations = { parent_fleet: Fleet };\n "list_children" if "viewer" on "parent_fleet";\n "read" if "viewer" on "parent_fleet";\n "modify" if "admin" on "parent_fleet";\n "create_child" if "admin" on "parent_fleet";\n }\n has_relation(fleet: Fleet, "parent_fleet", child: IpPool)\n if child.fleet = fleet;\n \n\n resource InstanceNetworkInterface {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n\n relations = {\n containing_project: Project,\n parent: Instance\n };\n "list_children" if "viewer" on "containing_project";\n "read" if "viewer" on "containing_project";\n "modify" if "collaborator" on "containing_project";\n "create_child" if "collaborator" on "containing_project";\n }\n\n has_relation(project: Project, "containing_project", child: InstanceNetworkInterface)\n if has_relation(project, "containing_project", child.instance);\n\n has_relation(parent: Instance, "parent", child: InstanceNetworkInterface)\n if child.instance = parent;\n \n\n resource Vpc {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n\n relations = { containing_project: Project };\n "list_children" if "viewer" on "containing_project";\n "read" if "viewer" on "containing_project";\n "modify" if "collaborator" on "containing_project";\n "create_child" if "collaborator" on "containing_project";\n }\n\n has_relation(parent: Project, "containing_project", child: Vpc)\n if child.project = parent;\n \n\n resource VpcRouter {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n\n relations = {\n containing_project: Project,\n parent: Vpc\n };\n "list_children" if "viewer" on "containing_project";\n "read" if "viewer" on "containing_project";\n "modify" if "collaborator" on "containing_project";\n "create_child" if "collaborator" on "containing_project";\n }\n\n has_relation(project: Project, "containing_project", child: VpcRouter)\n if has_relation(project, "containing_project", child.vpc);\n\n has_relation(parent: Vpc, "parent", child: VpcRouter)\n if child.vpc = parent;\n \n\n resource InternetGateway {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n\n relations = {\n containing_project: Project,\n parent: Vpc\n };\n "list_children" if "viewer" on "containing_project";\n "read" if "viewer" on "containing_project";\n "modify" if "collaborator" on "containing_project";\n "create_child" if "collaborator" on "containing_project";\n }\n\n has_relation(project: Project, "containing_project", child: InternetGateway)\n if has_relation(project, "containing_project", child.vpc);\n\n has_relation(parent: Vpc, "parent", child: InternetGateway)\n if child.vpc = parent;\n \n\n resource InternetGatewayIpPool {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n\n relations = {\n containing_project: Project,\n parent: InternetGateway\n };\n "list_children" if "viewer" on "containing_project";\n "read" if "viewer" on "containing_project";\n "modify" if "collaborator" on "containing_project";\n "create_child" if "collaborator" on "containing_project";\n }\n\n has_relation(project: Project, "containing_project", child: InternetGatewayIpPool)\n if has_relation(project, "containing_project", child.internet_gateway);\n\n has_relation(parent: InternetGateway, "parent", child: InternetGatewayIpPool)\n if child.internet_gateway = parent;\n \n\n resource InternetGatewayIpAddress {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n\n relations = {\n containing_project: Project,\n parent: InternetGateway\n };\n "list_children" if "viewer" on "containing_project";\n "read" if "viewer" on "containing_project";\n "modify" if "collaborator" on "containing_project";\n "create_child" if "collaborator" on "containing_project";\n }\n\n has_relation(project: Project, "containing_project", child: InternetGatewayIpAddress)\n if has_relation(project, "containing_project", child.internet_gateway);\n\n has_relation(parent: InternetGateway, "parent", child: InternetGatewayIpAddress)\n if child.internet_gateway = parent;\n \n\n resource RouterRoute {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n\n relations = {\n containing_project: Project,\n parent: VpcRouter\n };\n "list_children" if "viewer" on "containing_project";\n "read" if "viewer" on "containing_project";\n "modify" if "collaborator" on "containing_project";\n "create_child" if "collaborator" on "containing_project";\n }\n\n has_relation(project: Project, "containing_project", child: RouterRoute)\n if has_relation(project, "containing_project", child.vpc_router);\n\n has_relation(parent: VpcRouter, "parent", child: RouterRoute)\n if child.vpc_router = parent;\n \n\n resource VpcSubnet {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n\n relations = {\n containing_project: Project,\n parent: Vpc\n };\n "list_children" if "viewer" on "containing_project";\n "read" if "viewer" on "containing_project";\n "modify" if "collaborator" on "containing_project";\n "create_child" if "collaborator" on "containing_project";\n }\n\n has_relation(project: Project, "containing_project", child: VpcSubnet)\n if has_relation(project, "containing_project", child.vpc);\n\n has_relation(parent: Vpc, "parent", child: VpcSubnet)\n if child.vpc = parent;\n \n\n resource FloatingIp {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n\n relations = { containing_project: Project };\n "list_children" if "viewer" on "containing_project";\n "read" if "viewer" on "containing_project";\n "modify" if "collaborator" on "containing_project";\n "create_child" if "collaborator" on "containing_project";\n }\n\n has_relation(parent: Project, "containing_project", child: FloatingIp)\n if child.project = parent;\n \n\n resource Image {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n\n relations = { containing_silo: Silo };\n "list_children" if "viewer" on "containing_silo";\n "read" if "viewer" on "containing_silo";\n "modify" if "collaborator" on "containing_silo";\n "create_child" if "collaborator" on "containing_silo";\n }\n\n has_relation(parent: Silo, "containing_silo", child: Image)\n if child.silo = parent;\n \n\n resource SiloImage {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n\n relations = { containing_silo: Silo };\n "list_children" if "viewer" on "containing_silo";\n "read" if "viewer" on "containing_silo";\n "modify" if "collaborator" on "containing_silo";\n "create_child" if "collaborator" on "containing_silo";\n }\n\n has_relation(parent: Silo, "containing_silo", child: SiloImage)\n if child.silo = parent;\n \n\n resource AddressLot {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n \n relations = { parent_fleet: Fleet };\n "list_children" if "viewer" on "parent_fleet";\n "read" if "viewer" on "parent_fleet";\n "modify" if "admin" on "parent_fleet";\n "create_child" if "admin" on "parent_fleet";\n }\n has_relation(fleet: Fleet, "parent_fleet", child: AddressLot)\n if child.fleet = fleet;\n \n\n resource Blueprint {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n \n relations = { parent_fleet: Fleet };\n "list_children" if "viewer" on "parent_fleet";\n "read" if "viewer" on "parent_fleet";\n "modify" if "admin" on "parent_fleet";\n "create_child" if "admin" on "parent_fleet";\n }\n has_relation(fleet: Fleet, "parent_fleet", child: Blueprint)\n if child.fleet = fleet;\n \n\n resource LoopbackAddress {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n \n relations = { parent_fleet: Fleet };\n "list_children" if "viewer" on "parent_fleet";\n "read" if "viewer" on "parent_fleet";\n "modify" if "admin" on "parent_fleet";\n "create_child" if "admin" on "parent_fleet";\n }\n has_relation(fleet: Fleet, "parent_fleet", child: LoopbackAddress)\n if child.fleet = fleet;\n \n\n\n resource ConsoleSession {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n \n relations = { parent_fleet: Fleet };\n "list_children" if "viewer" on "parent_fleet";\n "read" if "viewer" on "parent_fleet";\n "modify" if "admin" on "parent_fleet";\n "create_child" if "admin" on "parent_fleet";\n }\n has_relation(fleet: Fleet, "parent_fleet", child: ConsoleSession)\n if child.fleet = fleet;\n \n\n resource DeviceAuthRequest {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n \n relations = { parent_fleet: Fleet };\n "list_children" if "viewer" on "parent_fleet";\n "read" if "viewer" on "parent_fleet";\n "modify" if "admin" on "parent_fleet";\n "create_child" if "admin" on "parent_fleet";\n }\n has_relation(fleet: Fleet, "parent_fleet", child: DeviceAuthRequest)\n if child.fleet = fleet;\n \n\n resource DeviceAccessToken {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n \n relations = { parent_fleet: Fleet };\n "list_children" if "viewer" on "parent_fleet";\n "read" if "viewer" on "parent_fleet";\n "modify" if "admin" on "parent_fleet";\n "create_child" if "admin" on "parent_fleet";\n }\n has_relation(fleet: Fleet, "parent_fleet", child: DeviceAccessToken)\n if child.fleet = fleet;\n \n\n resource PhysicalDisk {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n \n relations = { parent_fleet: Fleet };\n "list_children" if "viewer" on "parent_fleet";\n "read" if "viewer" on "parent_fleet";\n "modify" if "admin" on "parent_fleet";\n "create_child" if "admin" on "parent_fleet";\n }\n has_relation(fleet: Fleet, "parent_fleet", child: PhysicalDisk)\n if child.fleet = fleet;\n \n\n resource Rack {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n \n relations = { parent_fleet: Fleet };\n "list_children" if "viewer" on "parent_fleet";\n "read" if "viewer" on "parent_fleet";\n "modify" if "admin" on "parent_fleet";\n "create_child" if "admin" on "parent_fleet";\n }\n has_relation(fleet: Fleet, "parent_fleet", child: Rack)\n if child.fleet = fleet;\n \n\n resource RoleBuiltin {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n \n relations = { parent_fleet: Fleet };\n "list_children" if "viewer" on "parent_fleet";\n "read" if "viewer" on "parent_fleet";\n "modify" if "admin" on "parent_fleet";\n "create_child" if "admin" on "parent_fleet";\n }\n has_relation(fleet: Fleet, "parent_fleet", child: RoleBuiltin)\n if child.fleet = fleet;\n \n\n\n\n\n\n resource SupportBundle {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n \n relations = { parent_fleet: Fleet };\n "list_children" if "viewer" on "parent_fleet";\n "read" if "viewer" on "parent_fleet";\n "modify" if "admin" on "parent_fleet";\n "create_child" if "admin" on "parent_fleet";\n }\n has_relation(fleet: Fleet, "parent_fleet", child: SupportBundle)\n if child.fleet = fleet;\n \n\n\n\n resource Sled {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n \n relations = { parent_fleet: Fleet };\n "list_children" if "viewer" on "parent_fleet";\n "read" if "viewer" on "parent_fleet";\n "modify" if "admin" on "parent_fleet";\n "create_child" if "admin" on "parent_fleet";\n }\n has_relation(fleet: Fleet, "parent_fleet", child: Sled)\n if child.fleet = fleet;\n \n\n resource TufRepo {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n \n relations = { parent_fleet: Fleet };\n "list_children" if "viewer" on "parent_fleet";\n "read" if "viewer" on "parent_fleet";\n "modify" if "admin" on "parent_fleet";\n "create_child" if "admin" on "parent_fleet";\n }\n has_relation(fleet: Fleet, "parent_fleet", child: TufRepo)\n if child.fleet = fleet;\n \n\n resource TufArtifact {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n \n relations = { parent_fleet: Fleet };\n "list_children" if "viewer" on "parent_fleet";\n "read" if "viewer" on "parent_fleet";\n "modify" if "admin" on "parent_fleet";\n "create_child" if "admin" on "parent_fleet";\n }\n has_relation(fleet: Fleet, "parent_fleet", child: TufArtifact)\n if child.fleet = fleet;\n \n\n resource WebhookEvent {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n \n relations = { parent_fleet: Fleet };\n "list_children" if "viewer" on "parent_fleet";\n "read" if "viewer" on "parent_fleet";\n "modify" if "admin" on "parent_fleet";\n "create_child" if "admin" on "parent_fleet";\n }\n has_relation(fleet: Fleet, "parent_fleet", child: WebhookEvent)\n if child.fleet = fleet;\n \n\n resource WebhookReceiver {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n \n relations = { parent_fleet: Fleet };\n "list_children" if "viewer" on "parent_fleet";\n "read" if "viewer" on "parent_fleet";\n "modify" if "admin" on "parent_fleet";\n "create_child" if "admin" on "parent_fleet";\n }\n has_relation(fleet: Fleet, "parent_fleet", child: WebhookReceiver)\n if child.fleet = fleet;\n \n\n\n resource Zpool {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n \n relations = { parent_fleet: Fleet };\n "list_children" if "viewer" on "parent_fleet";\n "read" if "viewer" on "parent_fleet";\n "modify" if "admin" on "parent_fleet";\n "create_child" if "admin" on "parent_fleet";\n }\n has_relation(fleet: Fleet, "parent_fleet", child: Zpool)\n if child.fleet = fleet;\n \n\n resource Service {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n \n relations = { parent_fleet: Fleet };\n "list_children" if "viewer" on "parent_fleet";\n "read" if "viewer" on "parent_fleet";\n "modify" if "admin" on "parent_fleet";\n "create_child" if "admin" on "parent_fleet";\n }\n has_relation(fleet: Fleet, "parent_fleet", child: Service)\n if child.fleet = fleet;\n \n\n resource UserBuiltin {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n \n relations = { parent_fleet: Fleet };\n "list_children" if "viewer" on "parent_fleet";\n "read" if "viewer" on "parent_fleet";\n "modify" if "admin" on "parent_fleet";\n "create_child" if "admin" on "parent_fleet";\n }\n has_relation(fleet: Fleet, "parent_fleet", child: UserBuiltin)\n if child.fleet = fleet;\n
1032025-04-30T06:42:35.883ZINFOtest_snapshot_basic (ServerContext): Setting up resolver using DNS address: [::1]:62208
1042025-04-30T06:42:35.884ZINFOtest_snapshot_basic (DnsResolver): new DNS resolver addresses = [[::1]:62208]
1052025-04-30T06:42:35.884ZINFOtest_snapshot_basic (ServerContext): Setting up qorb database pool from a single host url = PostgresConfigWithUrl {\n url_raw: "postgresql://root@[::1]:43851/omicron?sslmode=disable",\n config: Config {\n user: Some(\n "root",\n ),\n password: None,\n dbname: Some(\n "omicron",\n ),\n options: None,\n application_name: None,\n ssl_mode: Disable,\n host: [\n Tcp(\n "::1",\n ),\n ],\n hostaddr: [],\n port: [\n 43851,\n ],\n connect_timeout: None,\n tcp_user_timeout: None,\n keepalives: true,\n keepalives_idle: 7200s,\n keepalives_interval: None,\n keepalives_retries: None,\n target_session_attrs: Any,\n channel_binding: Prefer,\n load_balance_hosts: Disable,\n },\n}
1062025-04-30T06:42:35.885ZDEBGtest_snapshot_basic (ServerContext): registered USDT probes
1072025-04-30T06:42:35.928ZINFOtest_snapshot_basic (nexus): Database schema version is up to date desired_version = 140.0.0 found_version = 140.0.0
1082025-04-30T06:42:36.026ZDEBGtest_snapshot_basic (clickhouse-client): registered USDT probes id = a104ba56-2af7-434b-98b7-deada7ca0af0
1092025-04-30T06:42:36.191ZDEBGtest_snapshot_basic (dropshot_internal): registered endpoint local_addr = 127.0.0.1:61468 method = GET path = /bgtasks versions = all
1102025-04-30T06:42:36.191ZDEBGtest_snapshot_basic (dropshot_internal): registered endpoint local_addr = 127.0.0.1:61468 method = POST path = /bgtasks/activate versions = all
1112025-04-30T06:42:36.191ZDEBGtest_snapshot_basic (dropshot_internal): registered endpoint local_addr = 127.0.0.1:61468 method = GET path = /bgtasks/view/{bgtask_name} versions = all
1122025-04-30T06:42:36.192ZDEBGtest_snapshot_basic (dropshot_internal): registered endpoint local_addr = 127.0.0.1:61468 method = GET path = /clickhouse/policy versions = all
1132025-04-30T06:42:36.192ZDEBGtest_snapshot_basic (dropshot_internal): registered endpoint local_addr = 127.0.0.1:61468 method = POST path = /clickhouse/policy versions = all
1142025-04-30T06:42:36.192ZDEBGtest_snapshot_basic (dropshot_internal): registered endpoint local_addr = 127.0.0.1:61468 method = POST path = /crucible/0/upstairs/{upstairs_id}/downstairs/{downstairs_id}/stop-request versions = all
1152025-04-30T06:42:36.192ZDEBGtest_snapshot_basic (dropshot_internal): registered endpoint local_addr = 127.0.0.1:61468 method = POST path = /crucible/0/upstairs/{upstairs_id}/downstairs/{downstairs_id}/stopped versions = all
1162025-04-30T06:42:36.192ZDEBGtest_snapshot_basic (dropshot_internal): registered endpoint local_addr = 127.0.0.1:61468 method = POST path = /crucible/0/upstairs/{upstairs_id}/repair/{repair_id}/progress versions = all
1172025-04-30T06:42:36.192ZDEBGtest_snapshot_basic (dropshot_internal): registered endpoint local_addr = 127.0.0.1:61468 method = POST path = /crucible/0/upstairs/{upstairs_id}/repair-finish versions = all
1182025-04-30T06:42:36.192ZDEBGtest_snapshot_basic (dropshot_internal): registered endpoint local_addr = 127.0.0.1:61468 method = POST path = /crucible/0/upstairs/{upstairs_id}/repair-start versions = all
1192025-04-30T06:42:36.192ZDEBGtest_snapshot_basic (dropshot_internal): registered endpoint local_addr = 127.0.0.1:61468 method = POST path = /demo-saga versions = all
1202025-04-30T06:42:36.192ZDEBGtest_snapshot_basic (dropshot_internal): registered endpoint local_addr = 127.0.0.1:61468 method = POST path = /demo-saga/{demo_saga_id}/complete versions = all
1212025-04-30T06:42:36.192ZDEBGtest_snapshot_basic (dropshot_internal): registered endpoint local_addr = 127.0.0.1:61468 method = GET path = /deployment/blueprints/all versions = all
1222025-04-30T06:42:36.192ZDEBGtest_snapshot_basic (dropshot_internal): registered endpoint local_addr = 127.0.0.1:61468 method = DELETE path = /deployment/blueprints/all/{blueprint_id} versions = all
1232025-04-30T06:42:36.192ZDEBGtest_snapshot_basic (dropshot_internal): registered endpoint local_addr = 127.0.0.1:61468 method = GET path = /deployment/blueprints/all/{blueprint_id} versions = all
1242025-04-30T06:42:36.192ZDEBGtest_snapshot_basic (dropshot_internal): registered endpoint local_addr = 127.0.0.1:61468 method = POST path = /deployment/blueprints/import versions = all
1252025-04-30T06:42:36.192ZDEBGtest_snapshot_basic (dropshot_internal): registered endpoint local_addr = 127.0.0.1:61468 method = POST path = /deployment/blueprints/regenerate versions = all
1262025-04-30T06:42:36.192ZDEBGtest_snapshot_basic (dropshot_internal): registered endpoint local_addr = 127.0.0.1:61468 method = GET path = /deployment/blueprints/target versions = all
1272025-04-30T06:42:36.192ZDEBGtest_snapshot_basic (dropshot_internal): registered endpoint local_addr = 127.0.0.1:61468 method = POST path = /deployment/blueprints/target versions = all
1282025-04-30T06:42:36.192ZDEBGtest_snapshot_basic (dropshot_internal): registered endpoint local_addr = 127.0.0.1:61468 method = PUT path = /deployment/blueprints/target/enabled versions = all
1292025-04-30T06:42:36.192ZDEBGtest_snapshot_basic (dropshot_internal): registered endpoint local_addr = 127.0.0.1:61468 method = POST path = /disk/{disk_id}/remove-read-only-parent versions = all
1302025-04-30T06:42:36.192ZDEBGtest_snapshot_basic (dropshot_internal): registered endpoint local_addr = 127.0.0.1:61468 method = PUT path = /disks/{disk_id} versions = all
1312025-04-30T06:42:36.192ZDEBGtest_snapshot_basic (dropshot_internal): registered endpoint local_addr = 127.0.0.1:61468 method = GET path = /experimental/v1/system/support-bundles versions = all
1322025-04-30T06:42:36.192ZDEBGtest_snapshot_basic (dropshot_internal): registered endpoint local_addr = 127.0.0.1:61468 method = POST path = /experimental/v1/system/support-bundles versions = all
1332025-04-30T06:42:36.192ZDEBGtest_snapshot_basic (dropshot_internal): registered endpoint local_addr = 127.0.0.1:61468 method = DELETE path = /experimental/v1/system/support-bundles/{support_bundle} versions = all
1342025-04-30T06:42:36.192ZDEBGtest_snapshot_basic (dropshot_internal): registered endpoint local_addr = 127.0.0.1:61468 method = GET path = /experimental/v1/system/support-bundles/{support_bundle} versions = all
1352025-04-30T06:42:36.192ZDEBGtest_snapshot_basic (dropshot_internal): registered endpoint local_addr = 127.0.0.1:61468 method = GET path = /experimental/v1/system/support-bundles/{support_bundle}/download versions = all
1362025-04-30T06:42:36.192ZDEBGtest_snapshot_basic (dropshot_internal): registered endpoint local_addr = 127.0.0.1:61468 method = HEAD path = /experimental/v1/system/support-bundles/{support_bundle}/download versions = all
1372025-04-30T06:42:36.192ZDEBGtest_snapshot_basic (dropshot_internal): registered endpoint local_addr = 127.0.0.1:61468 method = GET path = /experimental/v1/system/support-bundles/{support_bundle}/download/{file} versions = all
1382025-04-30T06:42:36.192ZDEBGtest_snapshot_basic (dropshot_internal): registered endpoint local_addr = 127.0.0.1:61468 method = HEAD path = /experimental/v1/system/support-bundles/{support_bundle}/download/{file} versions = all
1392025-04-30T06:42:36.192ZDEBGtest_snapshot_basic (dropshot_internal): registered endpoint local_addr = 127.0.0.1:61468 method = GET path = /experimental/v1/system/support-bundles/{support_bundle}/index versions = all
1402025-04-30T06:42:36.192ZDEBGtest_snapshot_basic (dropshot_internal): registered endpoint local_addr = 127.0.0.1:61468 method = POST path = /instances/{instance_id}/migrate versions = all
1412025-04-30T06:42:36.192ZDEBGtest_snapshot_basic (dropshot_internal): registered endpoint local_addr = 127.0.0.1:61468 method = POST path = /metrics/collectors versions = all
1422025-04-30T06:42:36.193ZDEBGtest_snapshot_basic (dropshot_internal): registered endpoint local_addr = 127.0.0.1:61468 method = GET path = /metrics/collectors/{collector_id}/producers versions = all
1432025-04-30T06:42:36.193ZDEBGtest_snapshot_basic (dropshot_internal): registered endpoint local_addr = 127.0.0.1:61468 method = POST path = /metrics/producers versions = all
1442025-04-30T06:42:36.193ZDEBGtest_snapshot_basic (dropshot_internal): registered endpoint local_addr = 127.0.0.1:61468 method = GET path = /mgs-updates versions = all
1452025-04-30T06:42:36.193ZDEBGtest_snapshot_basic (dropshot_internal): registered endpoint local_addr = 127.0.0.1:61468 method = GET path = /nat/ipv4/changeset/{from_gen} versions = all
1462025-04-30T06:42:36.193ZDEBGtest_snapshot_basic (dropshot_internal): registered endpoint local_addr = 127.0.0.1:61468 method = GET path = /oximeter/read-policy versions = all
1472025-04-30T06:42:36.193ZDEBGtest_snapshot_basic (dropshot_internal): registered endpoint local_addr = 127.0.0.1:61468 method = POST path = /oximeter/read-policy versions = all
1482025-04-30T06:42:36.193ZDEBGtest_snapshot_basic (dropshot_internal): registered endpoint local_addr = 127.0.0.1:61468 method = POST path = /physical-disk/expunge versions = all
1492025-04-30T06:42:36.193ZDEBGtest_snapshot_basic (dropshot_internal): registered endpoint local_addr = 127.0.0.1:61468 method = GET path = /probes/{sled} versions = all
1502025-04-30T06:42:36.193ZDEBGtest_snapshot_basic (dropshot_internal): registered endpoint local_addr = 127.0.0.1:61468 method = PUT path = /racks/{rack_id}/initialization-complete versions = all
1512025-04-30T06:42:36.193ZDEBGtest_snapshot_basic (dropshot_internal): registered endpoint local_addr = 127.0.0.1:61468 method = GET path = /sagas versions = all
1522025-04-30T06:42:36.193ZDEBGtest_snapshot_basic (dropshot_internal): registered endpoint local_addr = 127.0.0.1:61468 method = GET path = /sagas/{saga_id} versions = all
1532025-04-30T06:42:36.193ZDEBGtest_snapshot_basic (dropshot_internal): registered endpoint local_addr = 127.0.0.1:61468 method = GET path = /sled-agents/{sled_id} versions = all
1542025-04-30T06:42:36.193ZDEBGtest_snapshot_basic (dropshot_internal): registered endpoint local_addr = 127.0.0.1:61468 method = POST path = /sled-agents/{sled_id} versions = all
1552025-04-30T06:42:36.193ZDEBGtest_snapshot_basic (dropshot_internal): registered endpoint local_addr = 127.0.0.1:61468 method = POST path = /sled-agents/{sled_id}/firewall-rules-update versions = all
1562025-04-30T06:42:36.193ZDEBGtest_snapshot_basic (dropshot_internal): registered endpoint local_addr = 127.0.0.1:61468 method = POST path = /sleds/add versions = all
1572025-04-30T06:42:36.193ZDEBGtest_snapshot_basic (dropshot_internal): registered endpoint local_addr = 127.0.0.1:61468 method = POST path = /sleds/expunge versions = all
1582025-04-30T06:42:36.193ZDEBGtest_snapshot_basic (dropshot_internal): registered endpoint local_addr = 127.0.0.1:61468 method = GET path = /sleds/uninitialized versions = all
1592025-04-30T06:42:36.193ZDEBGtest_snapshot_basic (dropshot_internal): registered endpoint local_addr = 127.0.0.1:61468 method = PUT path = /switch/{switch_id} versions = all
1602025-04-30T06:42:36.193ZDEBGtest_snapshot_basic (dropshot_internal): registered endpoint local_addr = 127.0.0.1:61468 method = GET path = /v1/ping versions = all
1612025-04-30T06:42:36.193ZDEBGtest_snapshot_basic (dropshot_internal): registered endpoint local_addr = 127.0.0.1:61468 method = PUT path = /vmms/{propolis_id} versions = all
1622025-04-30T06:42:36.193ZDEBGtest_snapshot_basic (dropshot_internal): registered endpoint local_addr = 127.0.0.1:61468 method = POST path = /volume/{volume_id}/remove-read-only-parent versions = all
1632025-04-30T06:42:36.193ZINFOtest_snapshot_basic (dropshot_internal): listening local_addr = 127.0.0.1:61468
1642025-04-30T06:42:36.193ZDEBGtest_snapshot_basic (dropshot_internal): successfully registered DTrace USDT probes local_addr = 127.0.0.1:61468
1652025-04-30T06:42:36.194ZINFOtest_snapshot_basic (SEC): SEC running sec_id = 354a52d4-3078-4706-81ce-12c32f6bbeaa
1662025-04-30T06:42:36.195ZTRACtest_snapshot_basic (DataLoader): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
1672025-04-30T06:42:36.195ZDEBGtest_snapshot_basic (DataLoader): roles roles = RoleSet { roles: {} }
1682025-04-30T06:42:36.198ZDEBGtest_snapshot_basic (DataLoader): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
1692025-04-30T06:42:36.198ZDEBGtest_snapshot_basic (DataLoader): attempting to create built-in users
1702025-04-30T06:42:36.200ZTRACtest_snapshot_basic (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
1712025-04-30T06:42:36.200ZDEBGtest_snapshot_basic (DataLoader): roles roles = RoleSet { roles: {} }
1722025-04-30T06:42:36.200ZDEBGtest_snapshot_basic (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
1732025-04-30T06:42:36.201ZINFOtest_snapshot_basic (MgsUpdateDriver): starting MgsUpdateDriver
1742025-04-30T06:42:36.247ZTRACtest_snapshot_basic (dns): buffer buffer = Length: 79 (0x4f) bytes\n0000: 40 ef 01 00 00 01 00 00 00 00 00 01 10 5f 63 72 @............_cr\n0010: 75 63 69 62 6c 65 2d 70 61 6e 74 72 79 04 5f 74 ucible-pantry._t\n0020: 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 cp.control-plane\n0030: 05 6f 78 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 .oxide.internal.\n0040: 00 21 00 01 00 00 29 04 d0 00 00 00 00 00 00 .!....)........ kind = dns peer_addr = [::1]:57251 req_id = d51a8c44-b673-4571-b8b6-365455b32af9
1752025-04-30T06:42:36.247ZDEBGtest_snapshot_basic (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 16623,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_crucible-pantry._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_crucible-pantry._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 16,\n 95,\n 99,\n 114,\n 117,\n 99,\n 105,\n 98,\n 108,\n 101,\n 45,\n 112,\n 97,\n 110,\n 116,\n 114,\n 121,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:57251 req_id = d51a8c44-b673-4571-b8b6-365455b32af9
1762025-04-30T06:42:36.248ZERROtest_snapshot_basic (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 16623, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_crucible-pantry._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_crucible-pantry._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 16, 95, 99, 114, 117, 99, 105, 98, 108, 101, 45, 112, 97, 110, 116, 114, 121, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } SERVFAIL: server is not authoritative for name: "_crucible-pantry._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:57251 req_id = d51a8c44-b673-4571-b8b6-365455b32af9
1772025-04-30T06:42:36.325ZINFOtest_snapshot_basic (DataLoader): created 5 built-in users
1782025-04-30T06:42:36.331ZTRACtest_snapshot_basic (DataLoader): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
1792025-04-30T06:42:36.331ZDEBGtest_snapshot_basic (DataLoader): roles roles = RoleSet { roles: {} }
1802025-04-30T06:42:36.338ZDEBGtest_snapshot_basic (DataLoader): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
1812025-04-30T06:42:36.338ZDEBGtest_snapshot_basic (DataLoader): attempting to create built-in roles
1822025-04-30T06:42:36.338ZTRACtest_snapshot_basic (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
1832025-04-30T06:42:36.338ZDEBGtest_snapshot_basic (DataLoader): roles roles = RoleSet { roles: {} }
1842025-04-30T06:42:36.339ZDEBGtest_snapshot_basic (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
1852025-04-30T06:42:36.452ZINFOtest_snapshot_basic (DataLoader): created 10 built-in roles
1862025-04-30T06:42:36.452ZTRACtest_snapshot_basic (DataLoader): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
1872025-04-30T06:42:36.452ZDEBGtest_snapshot_basic (DataLoader): roles roles = RoleSet { roles: {} }
1882025-04-30T06:42:36.454ZDEBGtest_snapshot_basic (DataLoader): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
1892025-04-30T06:42:36.454ZDEBGtest_snapshot_basic (DataLoader): attempting to create built-in role assignments
1902025-04-30T06:42:36.454ZTRACtest_snapshot_basic (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
1912025-04-30T06:42:36.454ZDEBGtest_snapshot_basic (DataLoader): roles roles = RoleSet { roles: {} }
1922025-04-30T06:42:36.455ZDEBGtest_snapshot_basic (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
1932025-04-30T06:42:36.850ZINFOtest_snapshot_basic (DataLoader): created 4 built-in role assignments
1942025-04-30T06:42:36.850ZTRACtest_snapshot_basic (DataLoader): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
1952025-04-30T06:42:36.850ZDEBGtest_snapshot_basic (DataLoader): roles roles = RoleSet { roles: {} }
1962025-04-30T06:42:36.852ZDEBGtest_snapshot_basic (DataLoader): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
1972025-04-30T06:42:36.852ZDEBGtest_snapshot_basic (DataLoader): attempting to create built-in silos
1982025-04-30T06:42:36.852ZTRACtest_snapshot_basic (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
1992025-04-30T06:42:36.852ZDEBGtest_snapshot_basic (DataLoader): roles roles = RoleSet { roles: {} }
2002025-04-30T06:42:36.853ZDEBGtest_snapshot_basic (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
2012025-04-30T06:42:36.940ZINFOtest_snapshot_basic (DataLoader): created 2 built-in silos
2022025-04-30T06:42:36.940ZTRACtest_snapshot_basic (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
2032025-04-30T06:42:36.941ZDEBGtest_snapshot_basic (DataLoader): roles roles = RoleSet { roles: {} }
2042025-04-30T06:42:36.941ZDEBGtest_snapshot_basic (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
2052025-04-30T06:42:36.962ZTRACtest_snapshot_basic (DataLoader): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
2062025-04-30T06:42:36.963ZDEBGtest_snapshot_basic (DataLoader): roles roles = RoleSet { roles: {} }
2072025-04-30T06:42:36.965ZDEBGtest_snapshot_basic (DataLoader): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
2082025-04-30T06:42:36.965ZDEBGtest_snapshot_basic (DataLoader): attempting to create built-in projects
2092025-04-30T06:42:36.965ZTRACtest_snapshot_basic (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
2102025-04-30T06:42:36.965ZDEBGtest_snapshot_basic (DataLoader): roles roles = RoleSet { roles: {} }
2112025-04-30T06:42:36.965ZDEBGtest_snapshot_basic (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
2122025-04-30T06:42:36.970ZTRACtest_snapshot_basic (DataLoader): authorize begin action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }
2132025-04-30T06:42:36.970ZTRACtest_snapshot_basic (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-5110-4000-8000-000000000001 resource_type = Silo
2142025-04-30T06:42:36.971ZTRACtest_snapshot_basic (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
2152025-04-30T06:42:36.971ZDEBGtest_snapshot_basic (DataLoader): roles roles = RoleSet { roles: {} }
2162025-04-30T06:42:36.971ZDEBGtest_snapshot_basic (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
2172025-04-30T06:42:37.045ZTRACtest_snapshot_basic (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
2182025-04-30T06:42:37.045ZTRACtest_snapshot_basic (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
2192025-04-30T06:42:37.045ZDEBGtest_snapshot_basic (DataLoader): roles roles = RoleSet { roles: {} }
2202025-04-30T06:42:37.045ZDEBGtest_snapshot_basic (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
2212025-04-30T06:42:37.056ZDEBGtest_snapshot_basic (DataLoader): roles roles = RoleSet { roles: {} }
2222025-04-30T06:42:37.059ZDEBGtest_snapshot_basic (DataLoader): authorize result action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) } result = Ok(())
2232025-04-30T06:42:37.059ZTRACtest_snapshot_basic (DataLoader): authorize begin action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }
2242025-04-30T06:42:37.059ZTRACtest_snapshot_basic (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-5110-4000-8000-000000000001 resource_type = Silo
2252025-04-30T06:42:37.059ZTRACtest_snapshot_basic (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
2262025-04-30T06:42:37.059ZDEBGtest_snapshot_basic (DataLoader): roles roles = RoleSet { roles: {} }
2272025-04-30T06:42:37.060ZDEBGtest_snapshot_basic (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
2282025-04-30T06:42:37.062ZTRACtest_snapshot_basic (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
2292025-04-30T06:42:37.062ZTRACtest_snapshot_basic (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
2302025-04-30T06:42:37.062ZDEBGtest_snapshot_basic (DataLoader): roles roles = RoleSet { roles: {} }
2312025-04-30T06:42:37.062ZDEBGtest_snapshot_basic (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
2322025-04-30T06:42:37.064ZDEBGtest_snapshot_basic (DataLoader): roles roles = RoleSet { roles: {} }
2332025-04-30T06:42:37.067ZDEBGtest_snapshot_basic (DataLoader): authorize result action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) } result = Ok(())
2342025-04-30T06:42:37.067ZTRACtest_snapshot_basic (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
2352025-04-30T06:42:37.067ZDEBGtest_snapshot_basic (DataLoader): roles roles = RoleSet { roles: {} }
2362025-04-30T06:42:37.068ZDEBGtest_snapshot_basic (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
2372025-04-30T06:42:37.088ZINFOtest_snapshot_basic (DataLoader): created built-in services project
2382025-04-30T06:42:37.088ZTRACtest_snapshot_basic (DataLoader): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
2392025-04-30T06:42:37.089ZDEBGtest_snapshot_basic (DataLoader): roles roles = RoleSet { roles: {} }
2402025-04-30T06:42:37.092ZDEBGtest_snapshot_basic (DataLoader): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
2412025-04-30T06:42:37.092ZDEBGtest_snapshot_basic (DataLoader): attempting to create built-in VPCs
2422025-04-30T06:42:37.092ZTRACtest_snapshot_basic (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
2432025-04-30T06:42:37.092ZDEBGtest_snapshot_basic (DataLoader): roles roles = RoleSet { roles: {} }
2442025-04-30T06:42:37.092ZDEBGtest_snapshot_basic (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
2452025-04-30T06:42:37.100ZTRACtest_snapshot_basic (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
2462025-04-30T06:42:37.100ZDEBGtest_snapshot_basic (DataLoader): roles roles = RoleSet { roles: {} }
2472025-04-30T06:42:37.101ZDEBGtest_snapshot_basic (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
2482025-04-30T06:42:37.121ZTRACtest_snapshot_basic (DataLoader): authorize begin action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }
2492025-04-30T06:42:37.121ZTRACtest_snapshot_basic (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-4401-4000-8000-000000000000 resource_type = Project
2502025-04-30T06:42:37.121ZTRACtest_snapshot_basic (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
2512025-04-30T06:42:37.127ZDEBGtest_snapshot_basic (DataLoader): roles roles = RoleSet { roles: {} }
2522025-04-30T06:42:37.129ZDEBGtest_snapshot_basic (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
2532025-04-30T06:42:37.145ZTRACtest_snapshot_basic (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-5110-4000-8000-000000000001 resource_type = Silo
2542025-04-30T06:42:37.145ZTRACtest_snapshot_basic (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
2552025-04-30T06:42:37.145ZDEBGtest_snapshot_basic (DataLoader): roles roles = RoleSet { roles: {} }
2562025-04-30T06:42:37.146ZDEBGtest_snapshot_basic (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
2572025-04-30T06:42:37.152ZTRACtest_snapshot_basic (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
2582025-04-30T06:42:37.153ZTRACtest_snapshot_basic (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
2592025-04-30T06:42:37.153ZDEBGtest_snapshot_basic (DataLoader): roles roles = RoleSet { roles: {} }
2602025-04-30T06:42:37.153ZDEBGtest_snapshot_basic (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
2612025-04-30T06:42:37.158ZDEBGtest_snapshot_basic (DataLoader): roles roles = RoleSet { roles: {} }
2622025-04-30T06:42:37.164ZDEBGtest_snapshot_basic (DataLoader): authorize result action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) } result = Ok(())
2632025-04-30T06:42:37.164ZTRACtest_snapshot_basic (DataLoader): successful lookup of siloed resource "Project" using built-in user
2642025-04-30T06:42:37.164ZTRACtest_snapshot_basic (DataLoader): authorize begin action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }
2652025-04-30T06:42:37.164ZTRACtest_snapshot_basic (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-4401-4000-8000-000000000000 resource_type = Project
2662025-04-30T06:42:37.164ZTRACtest_snapshot_basic (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
2672025-04-30T06:42:37.164ZDEBGtest_snapshot_basic (DataLoader): roles roles = RoleSet { roles: {} }
2682025-04-30T06:42:37.164ZDEBGtest_snapshot_basic (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
2692025-04-30T06:42:37.170ZTRACtest_snapshot_basic (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-5110-4000-8000-000000000001 resource_type = Silo
2702025-04-30T06:42:37.170ZTRACtest_snapshot_basic (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
2712025-04-30T06:42:37.170ZDEBGtest_snapshot_basic (DataLoader): roles roles = RoleSet { roles: {} }
2722025-04-30T06:42:37.171ZDEBGtest_snapshot_basic (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
2732025-04-30T06:42:37.187ZTRACtest_snapshot_basic (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
2742025-04-30T06:42:37.187ZTRACtest_snapshot_basic (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
2752025-04-30T06:42:37.187ZDEBGtest_snapshot_basic (DataLoader): roles roles = RoleSet { roles: {} }
2762025-04-30T06:42:37.190ZDEBGtest_snapshot_basic (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
2772025-04-30T06:42:37.197ZDEBGtest_snapshot_basic (DataLoader): roles roles = RoleSet { roles: {} }
2782025-04-30T06:42:37.204ZDEBGtest_snapshot_basic (DataLoader): authorize result action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) } result = Ok(())
2792025-04-30T06:42:37.204ZTRACtest_snapshot_basic (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
2802025-04-30T06:42:37.204ZDEBGtest_snapshot_basic (DataLoader): roles roles = RoleSet { roles: {} }
2812025-04-30T06:42:37.205ZDEBGtest_snapshot_basic (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
2822025-04-30T06:42:37.244ZTRACtest_snapshot_basic (DataLoader): authorize begin action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ByName("oxide-services") }
2832025-04-30T06:42:37.244ZTRACtest_snapshot_basic (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-4401-4000-8000-000000000000 resource_type = Project
2842025-04-30T06:42:37.244ZTRACtest_snapshot_basic (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
2852025-04-30T06:42:37.244ZDEBGtest_snapshot_basic (DataLoader): roles roles = RoleSet { roles: {} }
2862025-04-30T06:42:37.245ZDEBGtest_snapshot_basic (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
2872025-04-30T06:42:37.251ZTRACtest_snapshot_basic (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-5110-4000-8000-000000000001 resource_type = Silo
2882025-04-30T06:42:37.251ZTRACtest_snapshot_basic (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
2892025-04-30T06:42:37.251ZDEBGtest_snapshot_basic (DataLoader): roles roles = RoleSet { roles: {} }
2902025-04-30T06:42:37.252ZDEBGtest_snapshot_basic (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
2912025-04-30T06:42:37.283ZTRACtest_snapshot_basic (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
2922025-04-30T06:42:37.283ZTRACtest_snapshot_basic (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
2932025-04-30T06:42:37.283ZDEBGtest_snapshot_basic (DataLoader): roles roles = RoleSet { roles: {} }
2942025-04-30T06:42:37.284ZDEBGtest_snapshot_basic (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
2952025-04-30T06:42:37.291ZDEBGtest_snapshot_basic (DataLoader): roles roles = RoleSet { roles: {} }
2962025-04-30T06:42:37.301ZDEBGtest_snapshot_basic (DataLoader): authorize result action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ByName("oxide-services") } result = Ok(())
2972025-04-30T06:42:37.301ZTRACtest_snapshot_basic (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
2982025-04-30T06:42:37.301ZDEBGtest_snapshot_basic (DataLoader): roles roles = RoleSet { roles: {} }
2992025-04-30T06:42:37.305ZDEBGtest_snapshot_basic (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
3002025-04-30T06:42:37.314ZTRACtest_snapshot_basic (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
3012025-04-30T06:42:37.314ZDEBGtest_snapshot_basic (DataLoader): roles roles = RoleSet { roles: {} }
3022025-04-30T06:42:37.314ZDEBGtest_snapshot_basic (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
3032025-04-30T06:42:37.334ZTRACtest_snapshot_basic (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
3042025-04-30T06:42:37.334ZDEBGtest_snapshot_basic (DataLoader): roles roles = RoleSet { roles: {} }
3052025-04-30T06:42:37.335ZDEBGtest_snapshot_basic (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
3062025-04-30T06:42:37.361ZTRACtest_snapshot_basic (DataLoader): authorize begin action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ByName("oxide-services") }
3072025-04-30T06:42:37.361ZTRACtest_snapshot_basic (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-4401-4000-8000-000000000000 resource_type = Project
3082025-04-30T06:42:37.361ZTRACtest_snapshot_basic (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
3092025-04-30T06:42:37.361ZDEBGtest_snapshot_basic (DataLoader): roles roles = RoleSet { roles: {} }
3102025-04-30T06:42:37.362ZDEBGtest_snapshot_basic (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
3112025-04-30T06:42:37.373ZTRACtest_snapshot_basic (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-5110-4000-8000-000000000001 resource_type = Silo
3122025-04-30T06:42:37.373ZTRACtest_snapshot_basic (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
3132025-04-30T06:42:37.373ZDEBGtest_snapshot_basic (DataLoader): roles roles = RoleSet { roles: {} }
3142025-04-30T06:42:37.374ZDEBGtest_snapshot_basic (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
3152025-04-30T06:42:37.381ZTRACtest_snapshot_basic (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
3162025-04-30T06:42:37.381ZTRACtest_snapshot_basic (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
3172025-04-30T06:42:37.381ZDEBGtest_snapshot_basic (DataLoader): roles roles = RoleSet { roles: {} }
3182025-04-30T06:42:37.387ZDEBGtest_snapshot_basic (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
3192025-04-30T06:42:37.392ZDEBGtest_snapshot_basic (DataLoader): roles roles = RoleSet { roles: {} }
3202025-04-30T06:42:37.400ZDEBGtest_snapshot_basic (DataLoader): authorize result action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ByName("oxide-services") } result = Ok(())
3212025-04-30T06:42:37.400ZTRACtest_snapshot_basic (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
3222025-04-30T06:42:37.400ZDEBGtest_snapshot_basic (DataLoader): roles roles = RoleSet { roles: {} }
3232025-04-30T06:42:37.401ZDEBGtest_snapshot_basic (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
3242025-04-30T06:42:37.460ZTRACtest_snapshot_basic (DataLoader): authorize begin action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = VpcRouter { parent: Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ByName("oxide-services") }, key: 001de000-074c-4000-8000-000000000001, lookup_type: ById(001de000-074c-4000-8000-000000000001) }
3252025-04-30T06:42:37.460ZTRACtest_snapshot_basic (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-4401-4000-8000-000000000000 resource_type = Project
3262025-04-30T06:42:37.460ZTRACtest_snapshot_basic (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
3272025-04-30T06:42:37.460ZDEBGtest_snapshot_basic (DataLoader): roles roles = RoleSet { roles: {} }
3282025-04-30T06:42:37.461ZDEBGtest_snapshot_basic (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
3292025-04-30T06:42:37.463ZTRACtest_snapshot_basic (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-5110-4000-8000-000000000001 resource_type = Silo
3302025-04-30T06:42:37.463ZTRACtest_snapshot_basic (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
3312025-04-30T06:42:37.463ZDEBGtest_snapshot_basic (DataLoader): roles roles = RoleSet { roles: {} }
3322025-04-30T06:42:37.464ZDEBGtest_snapshot_basic (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
3332025-04-30T06:42:37.466ZTRACtest_snapshot_basic (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
3342025-04-30T06:42:37.466ZTRACtest_snapshot_basic (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
3352025-04-30T06:42:37.466ZDEBGtest_snapshot_basic (DataLoader): roles roles = RoleSet { roles: {} }
3362025-04-30T06:42:37.466ZDEBGtest_snapshot_basic (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
3372025-04-30T06:42:37.468ZDEBGtest_snapshot_basic (DataLoader): roles roles = RoleSet { roles: {} }
3382025-04-30T06:42:37.474ZDEBGtest_snapshot_basic (DataLoader): authorize result action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = VpcRouter { parent: Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ByName("oxide-services") }, key: 001de000-074c-4000-8000-000000000001, lookup_type: ById(001de000-074c-4000-8000-000000000001) } result = Ok(())
3392025-04-30T06:42:37.474ZTRACtest_snapshot_basic (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
3402025-04-30T06:42:37.474ZDEBGtest_snapshot_basic (DataLoader): roles roles = RoleSet { roles: {} }
3412025-04-30T06:42:37.474ZDEBGtest_snapshot_basic (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
3422025-04-30T06:42:37.536ZTRACtest_snapshot_basic (DataLoader): authorize begin action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = VpcRouter { parent: Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ByName("oxide-services") }, key: 001de000-074c-4000-8000-000000000001, lookup_type: ById(001de000-074c-4000-8000-000000000001) }
3432025-04-30T06:42:37.536ZTRACtest_snapshot_basic (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-4401-4000-8000-000000000000 resource_type = Project
3442025-04-30T06:42:37.536ZTRACtest_snapshot_basic (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
3452025-04-30T06:42:37.536ZDEBGtest_snapshot_basic (DataLoader): roles roles = RoleSet { roles: {} }
3462025-04-30T06:42:37.537ZDEBGtest_snapshot_basic (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
3472025-04-30T06:42:37.543ZTRACtest_snapshot_basic (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-5110-4000-8000-000000000001 resource_type = Silo
3482025-04-30T06:42:37.543ZTRACtest_snapshot_basic (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
3492025-04-30T06:42:37.543ZDEBGtest_snapshot_basic (DataLoader): roles roles = RoleSet { roles: {} }
3502025-04-30T06:42:37.543ZDEBGtest_snapshot_basic (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
3512025-04-30T06:42:37.548ZTRACtest_snapshot_basic (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
3522025-04-30T06:42:37.549ZTRACtest_snapshot_basic (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
3532025-04-30T06:42:37.549ZDEBGtest_snapshot_basic (DataLoader): roles roles = RoleSet { roles: {} }
3542025-04-30T06:42:37.549ZDEBGtest_snapshot_basic (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
3552025-04-30T06:42:37.551ZDEBGtest_snapshot_basic (DataLoader): roles roles = RoleSet { roles: {} }
3562025-04-30T06:42:37.559ZDEBGtest_snapshot_basic (DataLoader): authorize result action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = VpcRouter { parent: Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ByName("oxide-services") }, key: 001de000-074c-4000-8000-000000000001, lookup_type: ById(001de000-074c-4000-8000-000000000001) } result = Ok(())
3572025-04-30T06:42:37.559ZTRACtest_snapshot_basic (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
3582025-04-30T06:42:37.559ZDEBGtest_snapshot_basic (DataLoader): roles roles = RoleSet { roles: {} }
3592025-04-30T06:42:37.560ZDEBGtest_snapshot_basic (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
3602025-04-30T06:42:37.566ZDEBGtest_snapshot_basic (DataLoader): attempting to create built-in VPC firewall rules
3612025-04-30T06:42:37.566ZTRACtest_snapshot_basic (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
3622025-04-30T06:42:37.566ZDEBGtest_snapshot_basic (DataLoader): roles roles = RoleSet { roles: {} }
3632025-04-30T06:42:37.567ZDEBGtest_snapshot_basic (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
3642025-04-30T06:42:37.570ZTRACtest_snapshot_basic (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
3652025-04-30T06:42:37.570ZDEBGtest_snapshot_basic (DataLoader): roles roles = RoleSet { roles: {} }
3662025-04-30T06:42:37.571ZDEBGtest_snapshot_basic (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
3672025-04-30T06:42:37.573ZTRACtest_snapshot_basic (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
3682025-04-30T06:42:37.573ZDEBGtest_snapshot_basic (DataLoader): roles roles = RoleSet { roles: {} }
3692025-04-30T06:42:37.573ZDEBGtest_snapshot_basic (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
3702025-04-30T06:42:37.576ZTRACtest_snapshot_basic (DataLoader): authorize begin action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ById(001de000-074c-4000-8000-000000000000) }
3712025-04-30T06:42:37.576ZTRACtest_snapshot_basic (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-4401-4000-8000-000000000000 resource_type = Project
3722025-04-30T06:42:37.576ZTRACtest_snapshot_basic (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
3732025-04-30T06:42:37.576ZDEBGtest_snapshot_basic (DataLoader): roles roles = RoleSet { roles: {} }
3742025-04-30T06:42:37.577ZDEBGtest_snapshot_basic (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
3752025-04-30T06:42:37.578ZTRACtest_snapshot_basic (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-5110-4000-8000-000000000001 resource_type = Silo
3762025-04-30T06:42:37.578ZTRACtest_snapshot_basic (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
3772025-04-30T06:42:37.578ZDEBGtest_snapshot_basic (DataLoader): roles roles = RoleSet { roles: {} }
3782025-04-30T06:42:37.579ZDEBGtest_snapshot_basic (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
3792025-04-30T06:42:37.581ZTRACtest_snapshot_basic (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
3802025-04-30T06:42:37.581ZTRACtest_snapshot_basic (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
3812025-04-30T06:42:37.581ZDEBGtest_snapshot_basic (DataLoader): roles roles = RoleSet { roles: {} }
3822025-04-30T06:42:37.581ZDEBGtest_snapshot_basic (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
3832025-04-30T06:42:37.583ZDEBGtest_snapshot_basic (DataLoader): roles roles = RoleSet { roles: {} }
3842025-04-30T06:42:37.588ZDEBGtest_snapshot_basic (DataLoader): authorize result action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ById(001de000-074c-4000-8000-000000000000) } result = Ok(())
3852025-04-30T06:42:37.588ZTRACtest_snapshot_basic (DataLoader): successful lookup of siloed resource "Vpc" using built-in user
3862025-04-30T06:42:37.588ZTRACtest_snapshot_basic (DataLoader): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ById(001de000-074c-4000-8000-000000000000) }
3872025-04-30T06:42:37.588ZTRACtest_snapshot_basic (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-4401-4000-8000-000000000000 resource_type = Project
3882025-04-30T06:42:37.588ZTRACtest_snapshot_basic (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
3892025-04-30T06:42:37.588ZDEBGtest_snapshot_basic (DataLoader): roles roles = RoleSet { roles: {} }
3902025-04-30T06:42:37.589ZDEBGtest_snapshot_basic (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
3912025-04-30T06:42:37.591ZTRACtest_snapshot_basic (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-5110-4000-8000-000000000001 resource_type = Silo
3922025-04-30T06:42:37.591ZTRACtest_snapshot_basic (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
3932025-04-30T06:42:37.591ZDEBGtest_snapshot_basic (DataLoader): roles roles = RoleSet { roles: {} }
3942025-04-30T06:42:37.591ZDEBGtest_snapshot_basic (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
3952025-04-30T06:42:37.593ZTRACtest_snapshot_basic (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
3962025-04-30T06:42:37.593ZTRACtest_snapshot_basic (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
3972025-04-30T06:42:37.593ZDEBGtest_snapshot_basic (DataLoader): roles roles = RoleSet { roles: {} }
3982025-04-30T06:42:37.594ZDEBGtest_snapshot_basic (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
3992025-04-30T06:42:37.595ZDEBGtest_snapshot_basic (DataLoader): roles roles = RoleSet { roles: {} }
4002025-04-30T06:42:37.609ZDEBGtest_snapshot_basic (DataLoader): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ById(001de000-074c-4000-8000-000000000000) } result = Ok(())
4012025-04-30T06:42:37.609ZTRACtest_snapshot_basic (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
4022025-04-30T06:42:37.609ZDEBGtest_snapshot_basic (DataLoader): roles roles = RoleSet { roles: {} }
4032025-04-30T06:42:37.610ZDEBGtest_snapshot_basic (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
4042025-04-30T06:42:37.620ZTRACtest_snapshot_basic (DataLoader): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ById(001de000-074c-4000-8000-000000000000) }
4052025-04-30T06:42:37.620ZTRACtest_snapshot_basic (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-4401-4000-8000-000000000000 resource_type = Project
4062025-04-30T06:42:37.620ZTRACtest_snapshot_basic (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
4072025-04-30T06:42:37.620ZDEBGtest_snapshot_basic (DataLoader): roles roles = RoleSet { roles: {} }
4082025-04-30T06:42:37.622ZDEBGtest_snapshot_basic (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
4092025-04-30T06:42:37.625ZTRACtest_snapshot_basic (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-5110-4000-8000-000000000001 resource_type = Silo
4102025-04-30T06:42:37.625ZTRACtest_snapshot_basic (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
4112025-04-30T06:42:37.625ZDEBGtest_snapshot_basic (DataLoader): roles roles = RoleSet { roles: {} }
4122025-04-30T06:42:37.626ZDEBGtest_snapshot_basic (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
4132025-04-30T06:42:37.630ZTRACtest_snapshot_basic (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
4142025-04-30T06:42:37.630ZTRACtest_snapshot_basic (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
4152025-04-30T06:42:37.630ZDEBGtest_snapshot_basic (DataLoader): roles roles = RoleSet { roles: {} }
4162025-04-30T06:42:37.631ZDEBGtest_snapshot_basic (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
4172025-04-30T06:42:37.640ZDEBGtest_snapshot_basic (DataLoader): roles roles = RoleSet { roles: {} }
4182025-04-30T06:42:37.647ZDEBGtest_snapshot_basic (DataLoader): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ById(001de000-074c-4000-8000-000000000000) } result = Ok(())
4192025-04-30T06:42:37.647ZTRACtest_snapshot_basic (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
4202025-04-30T06:42:37.647ZDEBGtest_snapshot_basic (DataLoader): roles roles = RoleSet { roles: {} }
4212025-04-30T06:42:37.648ZDEBGtest_snapshot_basic (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
4222025-04-30T06:42:37.757ZINFOtest_snapshot_basic (DataLoader): created built-in VPC firewall rules
4232025-04-30T06:42:37.758ZDEBGtest_snapshot_basic (DataLoader): attempting to create built-in VPC Subnets
4242025-04-30T06:42:37.758ZTRACtest_snapshot_basic (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
4252025-04-30T06:42:37.758ZDEBGtest_snapshot_basic (DataLoader): roles roles = RoleSet { roles: {} }
4262025-04-30T06:42:37.759ZDEBGtest_snapshot_basic (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
4272025-04-30T06:42:37.765ZTRACtest_snapshot_basic (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
4282025-04-30T06:42:37.766ZDEBGtest_snapshot_basic (DataLoader): roles roles = RoleSet { roles: {} }
4292025-04-30T06:42:37.767ZDEBGtest_snapshot_basic (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
4302025-04-30T06:42:37.770ZTRACtest_snapshot_basic (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
4312025-04-30T06:42:37.770ZDEBGtest_snapshot_basic (DataLoader): roles roles = RoleSet { roles: {} }
4322025-04-30T06:42:37.771ZDEBGtest_snapshot_basic (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
4332025-04-30T06:42:37.775ZTRACtest_snapshot_basic (DataLoader): authorize begin action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ById(001de000-074c-4000-8000-000000000000) }
4342025-04-30T06:42:37.775ZTRACtest_snapshot_basic (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-4401-4000-8000-000000000000 resource_type = Project
4352025-04-30T06:42:37.776ZTRACtest_snapshot_basic (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
4362025-04-30T06:42:37.776ZDEBGtest_snapshot_basic (DataLoader): roles roles = RoleSet { roles: {} }
4372025-04-30T06:42:37.778ZDEBGtest_snapshot_basic (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
4382025-04-30T06:42:37.785ZTRACtest_snapshot_basic (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-5110-4000-8000-000000000001 resource_type = Silo
4392025-04-30T06:42:37.785ZTRACtest_snapshot_basic (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
4402025-04-30T06:42:37.785ZDEBGtest_snapshot_basic (DataLoader): roles roles = RoleSet { roles: {} }
4412025-04-30T06:42:37.785ZDEBGtest_snapshot_basic (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
4422025-04-30T06:42:37.789ZTRACtest_snapshot_basic (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
4432025-04-30T06:42:37.790ZTRACtest_snapshot_basic (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
4442025-04-30T06:42:37.790ZDEBGtest_snapshot_basic (DataLoader): roles roles = RoleSet { roles: {} }
4452025-04-30T06:42:37.791ZDEBGtest_snapshot_basic (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
4462025-04-30T06:42:37.800ZDEBGtest_snapshot_basic (DataLoader): roles roles = RoleSet { roles: {} }
4472025-04-30T06:42:37.808ZDEBGtest_snapshot_basic (DataLoader): authorize result action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ById(001de000-074c-4000-8000-000000000000) } result = Ok(())
4482025-04-30T06:42:37.808ZTRACtest_snapshot_basic (DataLoader): successful lookup of siloed resource "Vpc" using built-in user
4492025-04-30T06:42:37.809ZTRACtest_snapshot_basic (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
4502025-04-30T06:42:37.809ZDEBGtest_snapshot_basic (DataLoader): roles roles = RoleSet { roles: {} }
4512025-04-30T06:42:37.809ZDEBGtest_snapshot_basic (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
4522025-04-30T06:42:37.819ZTRACtest_snapshot_basic (DataLoader): authorize begin action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ById(001de000-074c-4000-8000-000000000000) }
4532025-04-30T06:42:37.819ZTRACtest_snapshot_basic (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-4401-4000-8000-000000000000 resource_type = Project
4542025-04-30T06:42:37.819ZTRACtest_snapshot_basic (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
4552025-04-30T06:42:37.819ZDEBGtest_snapshot_basic (DataLoader): roles roles = RoleSet { roles: {} }
4562025-04-30T06:42:37.820ZDEBGtest_snapshot_basic (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
4572025-04-30T06:42:37.823ZTRACtest_snapshot_basic (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-5110-4000-8000-000000000001 resource_type = Silo
4582025-04-30T06:42:37.824ZTRACtest_snapshot_basic (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
4592025-04-30T06:42:37.824ZDEBGtest_snapshot_basic (DataLoader): roles roles = RoleSet { roles: {} }
4602025-04-30T06:42:37.824ZDEBGtest_snapshot_basic (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
4612025-04-30T06:42:37.833ZTRACtest_snapshot_basic (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
4622025-04-30T06:42:37.833ZTRACtest_snapshot_basic (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
4632025-04-30T06:42:37.833ZDEBGtest_snapshot_basic (DataLoader): roles roles = RoleSet { roles: {} }
4642025-04-30T06:42:37.833ZDEBGtest_snapshot_basic (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
4652025-04-30T06:42:37.844ZDEBGtest_snapshot_basic (DataLoader): roles roles = RoleSet { roles: {} }
4662025-04-30T06:42:37.851ZDEBGtest_snapshot_basic (DataLoader): authorize result action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ById(001de000-074c-4000-8000-000000000000) } result = Ok(())
4672025-04-30T06:42:37.861ZTRACtest_snapshot_basic (DataLoader): authorize begin action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = VpcRouter { parent: Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ByName("oxide-services") }, key: 001de000-074c-4000-8000-000000000001, lookup_type: ById(001de000-074c-4000-8000-000000000001) }
4682025-04-30T06:42:37.861ZTRACtest_snapshot_basic (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-4401-4000-8000-000000000000 resource_type = Project
4692025-04-30T06:42:37.861ZTRACtest_snapshot_basic (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
4702025-04-30T06:42:37.861ZDEBGtest_snapshot_basic (DataLoader): roles roles = RoleSet { roles: {} }
4712025-04-30T06:42:37.862ZDEBGtest_snapshot_basic (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
4722025-04-30T06:42:37.869ZTRACtest_snapshot_basic (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-5110-4000-8000-000000000001 resource_type = Silo
4732025-04-30T06:42:37.869ZTRACtest_snapshot_basic (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
4742025-04-30T06:42:37.869ZDEBGtest_snapshot_basic (DataLoader): roles roles = RoleSet { roles: {} }
4752025-04-30T06:42:37.871ZDEBGtest_snapshot_basic (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
4762025-04-30T06:42:37.879ZTRACtest_snapshot_basic (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
4772025-04-30T06:42:37.879ZTRACtest_snapshot_basic (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
4782025-04-30T06:42:37.879ZDEBGtest_snapshot_basic (DataLoader): roles roles = RoleSet { roles: {} }
4792025-04-30T06:42:37.880ZDEBGtest_snapshot_basic (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
4802025-04-30T06:42:37.885ZDEBGtest_snapshot_basic (DataLoader): roles roles = RoleSet { roles: {} }
4812025-04-30T06:42:37.895ZDEBGtest_snapshot_basic (DataLoader): authorize result action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = VpcRouter { parent: Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ByName("oxide-services") }, key: 001de000-074c-4000-8000-000000000001, lookup_type: ById(001de000-074c-4000-8000-000000000001) } result = Ok(())
4822025-04-30T06:42:37.895ZTRACtest_snapshot_basic (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
4832025-04-30T06:42:37.895ZDEBGtest_snapshot_basic (DataLoader): roles roles = RoleSet { roles: {} }
4842025-04-30T06:42:37.896ZDEBGtest_snapshot_basic (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
4852025-04-30T06:42:37.983ZTRACtest_snapshot_basic (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
4862025-04-30T06:42:37.983ZDEBGtest_snapshot_basic (DataLoader): roles roles = RoleSet { roles: {} }
4872025-04-30T06:42:37.984ZDEBGtest_snapshot_basic (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
4882025-04-30T06:42:37.988ZTRACtest_snapshot_basic (DataLoader): authorize begin action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ById(001de000-074c-4000-8000-000000000000) }
4892025-04-30T06:42:37.988ZTRACtest_snapshot_basic (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-4401-4000-8000-000000000000 resource_type = Project
4902025-04-30T06:42:37.988ZTRACtest_snapshot_basic (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
4912025-04-30T06:42:37.988ZDEBGtest_snapshot_basic (DataLoader): roles roles = RoleSet { roles: {} }
4922025-04-30T06:42:37.992ZDEBGtest_snapshot_basic (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
4932025-04-30T06:42:37.994ZTRACtest_snapshot_basic (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-5110-4000-8000-000000000001 resource_type = Silo
4942025-04-30T06:42:37.994ZTRACtest_snapshot_basic (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
4952025-04-30T06:42:37.994ZDEBGtest_snapshot_basic (DataLoader): roles roles = RoleSet { roles: {} }
4962025-04-30T06:42:37.996ZDEBGtest_snapshot_basic (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
4972025-04-30T06:42:38.001ZTRACtest_snapshot_basic (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
4982025-04-30T06:42:38.002ZTRACtest_snapshot_basic (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
4992025-04-30T06:42:38.002ZDEBGtest_snapshot_basic (DataLoader): roles roles = RoleSet { roles: {} }
5002025-04-30T06:42:38.002ZDEBGtest_snapshot_basic (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
5012025-04-30T06:42:38.004ZDEBGtest_snapshot_basic (DataLoader): roles roles = RoleSet { roles: {} }
5022025-04-30T06:42:38.010ZDEBGtest_snapshot_basic (DataLoader): authorize result action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ById(001de000-074c-4000-8000-000000000000) } result = Ok(())
5032025-04-30T06:42:38.017ZTRACtest_snapshot_basic (DataLoader): authorize begin action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = VpcRouter { parent: Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ByName("oxide-services") }, key: 001de000-074c-4000-8000-000000000001, lookup_type: ById(001de000-074c-4000-8000-000000000001) }
5042025-04-30T06:42:38.018ZTRACtest_snapshot_basic (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-4401-4000-8000-000000000000 resource_type = Project
5052025-04-30T06:42:38.019ZTRACtest_snapshot_basic (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
5062025-04-30T06:42:38.019ZDEBGtest_snapshot_basic (DataLoader): roles roles = RoleSet { roles: {} }
5072025-04-30T06:42:38.020ZDEBGtest_snapshot_basic (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
5082025-04-30T06:42:38.025ZTRACtest_snapshot_basic (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-5110-4000-8000-000000000001 resource_type = Silo
5092025-04-30T06:42:38.025ZTRACtest_snapshot_basic (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
5102025-04-30T06:42:38.025ZDEBGtest_snapshot_basic (DataLoader): roles roles = RoleSet { roles: {} }
5112025-04-30T06:42:38.026ZDEBGtest_snapshot_basic (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
5122025-04-30T06:42:38.030ZTRACtest_snapshot_basic (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
5132025-04-30T06:42:38.030ZTRACtest_snapshot_basic (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
5142025-04-30T06:42:38.030ZDEBGtest_snapshot_basic (DataLoader): roles roles = RoleSet { roles: {} }
5152025-04-30T06:42:38.031ZDEBGtest_snapshot_basic (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
5162025-04-30T06:42:38.034ZDEBGtest_snapshot_basic (DataLoader): roles roles = RoleSet { roles: {} }
5172025-04-30T06:42:38.044ZDEBGtest_snapshot_basic (DataLoader): authorize result action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = VpcRouter { parent: Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ByName("oxide-services") }, key: 001de000-074c-4000-8000-000000000001, lookup_type: ById(001de000-074c-4000-8000-000000000001) } result = Ok(())
5182025-04-30T06:42:38.044ZTRACtest_snapshot_basic (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
5192025-04-30T06:42:38.044ZDEBGtest_snapshot_basic (DataLoader): roles roles = RoleSet { roles: {} }
5202025-04-30T06:42:38.045ZDEBGtest_snapshot_basic (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
5212025-04-30T06:42:38.056ZTRACtest_snapshot_basic (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
5222025-04-30T06:42:38.056ZDEBGtest_snapshot_basic (DataLoader): roles roles = RoleSet { roles: {} }
5232025-04-30T06:42:38.058ZDEBGtest_snapshot_basic (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
5242025-04-30T06:42:38.061ZTRACtest_snapshot_basic (DataLoader): authorize begin action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ById(001de000-074c-4000-8000-000000000000) }
5252025-04-30T06:42:38.061ZTRACtest_snapshot_basic (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-4401-4000-8000-000000000000 resource_type = Project
5262025-04-30T06:42:38.061ZTRACtest_snapshot_basic (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
5272025-04-30T06:42:38.061ZDEBGtest_snapshot_basic (DataLoader): roles roles = RoleSet { roles: {} }
5282025-04-30T06:42:38.062ZDEBGtest_snapshot_basic (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
5292025-04-30T06:42:38.065ZTRACtest_snapshot_basic (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-5110-4000-8000-000000000001 resource_type = Silo
5302025-04-30T06:42:38.066ZTRACtest_snapshot_basic (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
5312025-04-30T06:42:38.066ZDEBGtest_snapshot_basic (DataLoader): roles roles = RoleSet { roles: {} }
5322025-04-30T06:42:38.067ZDEBGtest_snapshot_basic (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
5332025-04-30T06:42:38.087ZTRACtest_snapshot_basic (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
5342025-04-30T06:42:38.087ZTRACtest_snapshot_basic (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
5352025-04-30T06:42:38.087ZDEBGtest_snapshot_basic (DataLoader): roles roles = RoleSet { roles: {} }
5362025-04-30T06:42:38.089ZDEBGtest_snapshot_basic (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
5372025-04-30T06:42:38.096ZDEBGtest_snapshot_basic (DataLoader): roles roles = RoleSet { roles: {} }
5382025-04-30T06:42:38.115ZDEBGtest_snapshot_basic (DataLoader): authorize result action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ById(001de000-074c-4000-8000-000000000000) } result = Ok(())
5392025-04-30T06:42:38.120ZTRACtest_snapshot_basic (DataLoader): authorize begin action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = VpcRouter { parent: Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ByName("oxide-services") }, key: 001de000-074c-4000-8000-000000000001, lookup_type: ById(001de000-074c-4000-8000-000000000001) }
5402025-04-30T06:42:38.120ZTRACtest_snapshot_basic (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-4401-4000-8000-000000000000 resource_type = Project
5412025-04-30T06:42:38.120ZTRACtest_snapshot_basic (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
5422025-04-30T06:42:38.120ZDEBGtest_snapshot_basic (DataLoader): roles roles = RoleSet { roles: {} }
5432025-04-30T06:42:38.121ZDEBGtest_snapshot_basic (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
5442025-04-30T06:42:38.129ZTRACtest_snapshot_basic (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-5110-4000-8000-000000000001 resource_type = Silo
5452025-04-30T06:42:38.129ZTRACtest_snapshot_basic (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
5462025-04-30T06:42:38.129ZDEBGtest_snapshot_basic (DataLoader): roles roles = RoleSet { roles: {} }
5472025-04-30T06:42:38.131ZDEBGtest_snapshot_basic (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
5482025-04-30T06:42:38.136ZTRACtest_snapshot_basic (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
5492025-04-30T06:42:38.136ZTRACtest_snapshot_basic (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
5502025-04-30T06:42:38.136ZDEBGtest_snapshot_basic (DataLoader): roles roles = RoleSet { roles: {} }
5512025-04-30T06:42:38.137ZDEBGtest_snapshot_basic (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
5522025-04-30T06:42:38.140ZDEBGtest_snapshot_basic (DataLoader): roles roles = RoleSet { roles: {} }
5532025-04-30T06:42:38.151ZDEBGtest_snapshot_basic (DataLoader): authorize result action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = VpcRouter { parent: Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ByName("oxide-services") }, key: 001de000-074c-4000-8000-000000000001, lookup_type: ById(001de000-074c-4000-8000-000000000001) } result = Ok(())
5542025-04-30T06:42:38.151ZTRACtest_snapshot_basic (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
5552025-04-30T06:42:38.151ZDEBGtest_snapshot_basic (DataLoader): roles roles = RoleSet { roles: {} }
5562025-04-30T06:42:38.152ZDEBGtest_snapshot_basic (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
5572025-04-30T06:42:38.158ZTRACtest_snapshot_basic (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
5582025-04-30T06:42:38.158ZDEBGtest_snapshot_basic (DataLoader): roles roles = RoleSet { roles: {} }
5592025-04-30T06:42:38.159ZDEBGtest_snapshot_basic (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
5602025-04-30T06:42:38.178ZINFOtest_snapshot_basic (DataLoader): created built-in services vpc subnets
5612025-04-30T06:42:38.178ZINFOtest_snapshot_basic (DataLoader): created built-in services vpc
5622025-04-30T06:42:38.178ZTRACtest_snapshot_basic (DataLoader): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
5632025-04-30T06:42:38.178ZDEBGtest_snapshot_basic (DataLoader): roles roles = RoleSet { roles: {} }
5642025-04-30T06:42:38.183ZDEBGtest_snapshot_basic (DataLoader): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
5652025-04-30T06:42:38.183ZDEBGtest_snapshot_basic (DataLoader): attempting to create silo users
5662025-04-30T06:42:38.183ZTRACtest_snapshot_basic (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
5672025-04-30T06:42:38.183ZDEBGtest_snapshot_basic (DataLoader): roles roles = RoleSet { roles: {} }
5682025-04-30T06:42:38.186ZDEBGtest_snapshot_basic (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
5692025-04-30T06:42:38.194ZINFOtest_snapshot_basic (DataLoader): created 2 silo users
5702025-04-30T06:42:38.194ZDEBGtest_snapshot_basic (DataLoader): attempting to create silo user role assignments
5712025-04-30T06:42:38.194ZTRACtest_snapshot_basic (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
5722025-04-30T06:42:38.194ZDEBGtest_snapshot_basic (DataLoader): roles roles = RoleSet { roles: {} }
5732025-04-30T06:42:38.195ZDEBGtest_snapshot_basic (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
5742025-04-30T06:42:38.199ZINFOtest_snapshot_basic (DataLoader): created 2 silo user role assignments
5752025-04-30T06:42:38.199ZTRACtest_snapshot_basic (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
5762025-04-30T06:42:38.199ZDEBGtest_snapshot_basic (DataLoader): roles roles = RoleSet { roles: {} }
5772025-04-30T06:42:38.199ZDEBGtest_snapshot_basic (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
5782025-04-30T06:42:38.211ZTRACtest_snapshot_basic (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
5792025-04-30T06:42:38.211ZDEBGtest_snapshot_basic (DataLoader): roles roles = RoleSet { roles: {} }
5802025-04-30T06:42:38.212ZDEBGtest_snapshot_basic (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
5812025-04-30T06:42:38.221ZTRACtest_snapshot_basic (DataLoader): authorize begin action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = IpPoolList
5822025-04-30T06:42:38.221ZTRACtest_snapshot_basic (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
5832025-04-30T06:42:38.221ZTRACtest_snapshot_basic (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
5842025-04-30T06:42:38.221ZDEBGtest_snapshot_basic (DataLoader): roles roles = RoleSet { roles: {} }
5852025-04-30T06:42:38.222ZDEBGtest_snapshot_basic (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
5862025-04-30T06:42:38.230ZDEBGtest_snapshot_basic (DataLoader): roles roles = RoleSet { roles: {} }
5872025-04-30T06:42:38.233ZDEBGtest_snapshot_basic (DataLoader): authorize result action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = IpPoolList result = Ok(())
5882025-04-30T06:42:38.233ZTRACtest_snapshot_basic (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
5892025-04-30T06:42:38.233ZDEBGtest_snapshot_basic (DataLoader): roles roles = RoleSet { roles: {} }
5902025-04-30T06:42:38.235ZDEBGtest_snapshot_basic (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
5912025-04-30T06:42:38.244ZTRACtest_snapshot_basic (DataLoader): authorize begin action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = IpPoolList
5922025-04-30T06:42:38.244ZTRACtest_snapshot_basic (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
5932025-04-30T06:42:38.244ZTRACtest_snapshot_basic (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
5942025-04-30T06:42:38.244ZDEBGtest_snapshot_basic (DataLoader): roles roles = RoleSet { roles: {} }
5952025-04-30T06:42:38.245ZDEBGtest_snapshot_basic (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
5962025-04-30T06:42:38.256ZDEBGtest_snapshot_basic (DataLoader): roles roles = RoleSet { roles: {} }
5972025-04-30T06:42:38.259ZDEBGtest_snapshot_basic (DataLoader): authorize result action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = IpPoolList result = Ok(())
5982025-04-30T06:42:38.259ZTRACtest_snapshot_basic (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
5992025-04-30T06:42:38.259ZDEBGtest_snapshot_basic (DataLoader): roles roles = RoleSet { roles: {} }
6002025-04-30T06:42:38.259ZDEBGtest_snapshot_basic (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
6012025-04-30T06:42:38.337ZTRACtest_snapshot_basic (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
6022025-04-30T06:42:38.339ZDEBGtest_snapshot_basic (DataLoader): roles roles = RoleSet { roles: {} }
6032025-04-30T06:42:38.339ZDEBGtest_snapshot_basic (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
6042025-04-30T06:42:38.345ZINFOtest_snapshot_basic (nexus): populate complete
6052025-04-30T06:42:38.345ZINFOtest_snapshot_basic (nexus): activating background tasks
6062025-04-30T06:42:38.351ZDEBGtest_snapshot_basic (init_with_steps): Running step start_sled1
6072025-04-30T06:42:38.351ZINFOtest_snapshot_basic (omicron_sled_agent::sim::Server): setting up sled agent server sled_id = b6d65341-167c-41df-9b5c-41cded99c229
6082025-04-30T06:42:38.413ZINFOtest_snapshot_basic (SledAgent): created simulated sled agent server = b6d65341-167c-41df-9b5c-41cded99c229 sim_mode = Explicit sled_id = b6d65341-167c-41df-9b5c-41cded99c229
6092025-04-30T06:42:38.413ZINFOtest_snapshot_basic (omicron_sled_agent::sim::SimulatedUpstairs): registered sled b6d65341-167c-41df-9b5c-41cded99c229 storage
6102025-04-30T06:42:38.413ZDEBGtest_snapshot_basic (BackgroundTasks): activating background_task = dns_config_internal dns_group = internal iteration = 1 reason = Timeout
6112025-04-30T06:42:38.413ZDEBGtest_snapshot_basic (BackgroundTasks): reading DNS version background_task = dns_config_internal dns_group = internal
6122025-04-30T06:42:38.414ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database
6132025-04-30T06:42:38.414ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
6142025-04-30T06:42:38.415ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
6152025-04-30T06:42:38.415ZDEBGtest_snapshot_basic (BackgroundTasks): activating background_task = dns_servers_internal dns_group = internal iteration = 1 reason = Timeout
6162025-04-30T06:42:38.415ZTRACtest_snapshot_basic (DnsResolver): lookup_all_socket_v6 srv dns_name = _nameservice._tcp.control-plane.oxide.internal
6172025-04-30T06:42:38.417ZDEBGtest_snapshot_basic (BackgroundTasks): activating background_task = dns_propagation_internal dns_group = internal iteration = 1 reason = Timeout
6182025-04-30T06:42:38.417ZWARNtest_snapshot_basic (BackgroundTasks): DNS propagation: skipped background_task = dns_propagation_internal dns_group = internal reason = no config nor servers
6192025-04-30T06:42:38.417ZDEBGtest_snapshot_basic (BackgroundTasks): activation complete background_task = dns_propagation_internal dns_group = internal elapsed = 219.958\u{b5}s iteration = 1 status = {"error":"no config nor servers"}
6202025-04-30T06:42:38.417ZDEBGtest_snapshot_basic (BackgroundTasks): activating background_task = dns_config_external dns_group = external iteration = 1 reason = Timeout
6212025-04-30T06:42:38.417ZDEBGtest_snapshot_basic (BackgroundTasks): reading DNS version background_task = dns_config_external dns_group = external
6222025-04-30T06:42:38.417ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database
6232025-04-30T06:42:38.417ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
6242025-04-30T06:42:38.419ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
6252025-04-30T06:42:38.419ZDEBGtest_snapshot_basic (BackgroundTasks): activating background_task = dns_servers_external dns_group = external iteration = 1 reason = Timeout
6262025-04-30T06:42:38.419ZTRACtest_snapshot_basic (DnsResolver): lookup_all_socket_v6 srv dns_name = _external-dns._tcp.control-plane.oxide.internal
6272025-04-30T06:42:38.419ZDEBGtest_snapshot_basic (BackgroundTasks): activating background_task = dns_propagation_external dns_group = external iteration = 1 reason = Timeout
6282025-04-30T06:42:38.419ZWARNtest_snapshot_basic (BackgroundTasks): DNS propagation: skipped background_task = dns_propagation_external dns_group = external reason = no config nor servers
6292025-04-30T06:42:38.419ZDEBGtest_snapshot_basic (BackgroundTasks): activation complete background_task = dns_propagation_external dns_group = external elapsed = 26.002\u{b5}s iteration = 1 status = {"error":"no config nor servers"}
6302025-04-30T06:42:38.419ZDEBGtest_snapshot_basic (BackgroundTasks): activating background_task = metrics_producer_gc iteration = 1 reason = Timeout
6312025-04-30T06:42:38.419ZINFOtest_snapshot_basic (BackgroundTasks): Metric producer GC running background_task = metrics_producer_gc expiration = 2025-04-30 06:32:38.419484759 UTC
6322025-04-30T06:42:38.420ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = metrics_producer_gc resource = Database
6332025-04-30T06:42:38.420ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = metrics_producer_gc roles = RoleSet { roles: {} }
6342025-04-30T06:42:38.421ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = metrics_producer_gc resource = Database result = Ok(())
6352025-04-30T06:42:38.421ZDEBGtest_snapshot_basic (BackgroundTasks): activating background_task = external_endpoints iteration = 1 reason = Timeout
6362025-04-30T06:42:38.421ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Fleet
6372025-04-30T06:42:38.421ZTRACtest_snapshot_basic (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = external_endpoints resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
6382025-04-30T06:42:38.421ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database
6392025-04-30T06:42:38.421ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
6402025-04-30T06:42:38.421ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
6412025-04-30T06:42:38.421ZDEBGtest_snapshot_basic (BackgroundTasks): activating background_task = nat_v4_garbage_collector iteration = 1 reason = Timeout
6422025-04-30T06:42:38.421ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database
6432025-04-30T06:42:38.421ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
6442025-04-30T06:42:38.422ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
6452025-04-30T06:42:38.422ZDEBGtest_snapshot_basic (BackgroundTasks): activating background_task = bfd_manager iteration = 1 reason = Timeout
6462025-04-30T06:42:38.422ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = bfd_manager resource = Database
6472025-04-30T06:42:38.422ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = bfd_manager roles = RoleSet { roles: {} }
6482025-04-30T06:42:38.423ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = bfd_manager resource = Database result = Ok(())
6492025-04-30T06:42:38.423ZDEBGtest_snapshot_basic (BackgroundTasks): activating background_task = phantom_disks iteration = 1 reason = Timeout
6502025-04-30T06:42:38.423ZDEBGtest_snapshot_basic (BackgroundTasks): activating background_task = blueprint_loader iteration = 1 reason = Timeout
6512025-04-30T06:42:38.423ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig
6522025-04-30T06:42:38.423ZTRACtest_snapshot_basic (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = blueprint_loader resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
6532025-04-30T06:42:38.423ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database
6542025-04-30T06:42:38.423ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
6552025-04-30T06:42:38.424ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
6562025-04-30T06:42:38.424ZDEBGtest_snapshot_basic (BackgroundTasks): activating background_task = blueprint_executor iteration = 1 reason = Timeout
6572025-04-30T06:42:38.424ZWARNtest_snapshot_basic (BackgroundTasks): Blueprint execution: skipped background_task = blueprint_executor reason = no blueprint
6582025-04-30T06:42:38.424ZDEBGtest_snapshot_basic (BackgroundTasks): activation complete background_task = blueprint_executor elapsed = 240.699\u{b5}s iteration = 1 status = {"error":"no blueprint"}
6592025-04-30T06:42:38.424ZDEBGtest_snapshot_basic (BackgroundTasks): activating background_task = crdb_node_id_collector iteration = 1 reason = Timeout
6602025-04-30T06:42:38.424ZWARNtest_snapshot_basic (BackgroundTasks): Blueprint execution: skipped background_task = crdb_node_id_collector reason = no blueprint
6612025-04-30T06:42:38.424ZDEBGtest_snapshot_basic (BackgroundTasks): activation complete background_task = crdb_node_id_collector elapsed = 83.256\u{b5}s iteration = 1 status = {"error":"no blueprint"}
6622025-04-30T06:42:38.424ZDEBGtest_snapshot_basic (BackgroundTasks): activating background_task = inventory_collection iteration = 1 reason = Timeout
6632025-04-30T06:42:38.424ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
6642025-04-30T06:42:38.424ZTRACtest_snapshot_basic (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
6652025-04-30T06:42:38.424ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
6662025-04-30T06:42:38.424ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
6672025-04-30T06:42:38.425ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
6682025-04-30T06:42:38.425ZDEBGtest_snapshot_basic (BackgroundTasks): activating background_task = support_bundle_collector iteration = 1 reason = Timeout
6692025-04-30T06:42:38.426ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Fleet
6702025-04-30T06:42:38.426ZTRACtest_snapshot_basic (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = support_bundle_collector resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
6712025-04-30T06:42:38.426ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database
6722025-04-30T06:42:38.426ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
6732025-04-30T06:42:38.426ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
6742025-04-30T06:42:38.426ZDEBGtest_snapshot_basic (BackgroundTasks): activating background_task = physical_disk_adoption iteration = 1 reason = Timeout
6752025-04-30T06:42:38.426ZDEBGtest_snapshot_basic (BackgroundTasks): activation complete background_task = physical_disk_adoption elapsed = 20.18\u{b5}s iteration = 1 status = {"error":"task disabled"}
6762025-04-30T06:42:38.426ZDEBGtest_snapshot_basic (BackgroundTasks): activating background_task = blueprint_rendezvous iteration = 1 reason = Timeout
6772025-04-30T06:42:38.426ZWARNtest_snapshot_basic (BackgroundTasks): Blueprint rendezvous: skipped background_task = blueprint_rendezvous reason = no blueprint
6782025-04-30T06:42:38.426ZDEBGtest_snapshot_basic (BackgroundTasks): activation complete background_task = blueprint_rendezvous elapsed = 16.252\u{b5}s iteration = 1 status = {"error":"no blueprint"}
6792025-04-30T06:42:38.426ZDEBGtest_snapshot_basic (BackgroundTasks): activating background_task = decommissioned_disk_cleaner iteration = 1 reason = Timeout
6802025-04-30T06:42:38.426ZINFOtest_snapshot_basic (BackgroundTasks): Decommissioned Disk Cleaner disabled background_task = decommissioned_disk_cleaner
6812025-04-30T06:42:38.426ZDEBGtest_snapshot_basic (BackgroundTasks): activation complete background_task = decommissioned_disk_cleaner elapsed = 143.015\u{b5}s iteration = 1 status = {"deleted":0,"error":null,"error_count":0,"found":0,"not_ready_to_be_deleted":0}
6822025-04-30T06:42:38.427ZDEBGtest_snapshot_basic (BackgroundTasks): activating background_task = service_zone_nat_tracker iteration = 1 reason = Timeout
6832025-04-30T06:42:38.427ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Inventory
6842025-04-30T06:42:38.427ZTRACtest_snapshot_basic (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = service_zone_nat_tracker resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
6852025-04-30T06:42:38.427ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database
6862025-04-30T06:42:38.427ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
6872025-04-30T06:42:38.428ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
6882025-04-30T06:42:38.428ZDEBGtest_snapshot_basic (BackgroundTasks): activating background_task = switch_port_config_manager iteration = 1 reason = Timeout
6892025-04-30T06:42:38.428ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Fleet
6902025-04-30T06:42:38.428ZTRACtest_snapshot_basic (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = switch_port_config_manager resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
6912025-04-30T06:42:38.428ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database
6922025-04-30T06:42:38.428ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
6932025-04-30T06:42:38.429ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
6942025-04-30T06:42:38.431ZINFOtest_snapshot_basic (ArtifactStore): No ledger in /var/tmp/omicron_tmp/.tmp7W42qF/artifact-config.json server = b6d65341-167c-41df-9b5c-41cded99c229 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
6952025-04-30T06:42:38.431ZDEBGtest_snapshot_basic (ArtifactStore): Failed to read ledger: Not found in storage path = /var/tmp/omicron_tmp/.tmp7W42qF/artifact-config.json server = b6d65341-167c-41df-9b5c-41cded99c229 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
6962025-04-30T06:42:38.431ZINFOtest_snapshot_basic (ArtifactStore): No ledger in /var/tmp/omicron_tmp/.tmpaLyh3W/artifact-config.json server = b6d65341-167c-41df-9b5c-41cded99c229 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
6972025-04-30T06:42:38.431ZDEBGtest_snapshot_basic (ArtifactStore): Failed to read ledger: Not found in storage path = /var/tmp/omicron_tmp/.tmpaLyh3W/artifact-config.json server = b6d65341-167c-41df-9b5c-41cded99c229 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
6982025-04-30T06:42:38.486ZDEBGtest_snapshot_basic (dropshot (Repo Depot)): registered endpoint local_addr = [::1]:46799 method = GET path = /artifact/sha256/{sha256} server = b6d65341-167c-41df-9b5c-41cded99c229 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
6992025-04-30T06:42:38.486ZINFOtest_snapshot_basic (dropshot (Repo Depot)): listening local_addr = [::1]:46799 server = b6d65341-167c-41df-9b5c-41cded99c229 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
7002025-04-30T06:42:38.487ZDEBGtest_snapshot_basic (dropshot (Repo Depot)): successfully registered DTrace USDT probes local_addr = [::1]:46799 server = b6d65341-167c-41df-9b5c-41cded99c229 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
7012025-04-30T06:42:38.500ZDEBGtest_snapshot_basic (dropshot): registered endpoint local_addr = [::1]:41923 method = GET path = /artifacts sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
7022025-04-30T06:42:38.500ZDEBGtest_snapshot_basic (dropshot): registered endpoint local_addr = [::1]:41923 method = PUT path = /artifacts/{sha256} sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
7032025-04-30T06:42:38.500ZDEBGtest_snapshot_basic (dropshot): registered endpoint local_addr = [::1]:41923 method = POST path = /artifacts/{sha256}/copy-from-depot sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
7042025-04-30T06:42:38.500ZDEBGtest_snapshot_basic (dropshot): registered endpoint local_addr = [::1]:41923 method = GET path = /artifacts-config sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
7052025-04-30T06:42:38.500ZDEBGtest_snapshot_basic (dropshot): registered endpoint local_addr = [::1]:41923 method = PUT path = /artifacts-config sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
7062025-04-30T06:42:38.500ZDEBGtest_snapshot_basic (dropshot): registered endpoint local_addr = [::1]:41923 method = POST path = /boot-disk/{boot_disk}/os/write sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
7072025-04-30T06:42:38.500ZDEBGtest_snapshot_basic (dropshot): registered endpoint local_addr = [::1]:41923 method = GET path = /boot-disk/{boot_disk}/os/write/status sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
7082025-04-30T06:42:38.500ZDEBGtest_snapshot_basic (dropshot): registered endpoint local_addr = [::1]:41923 method = DELETE path = /boot-disk/{boot_disk}/os/write/status/{update_id} sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
7092025-04-30T06:42:38.500ZDEBGtest_snapshot_basic (dropshot): registered endpoint local_addr = [::1]:41923 method = GET path = /bootstore/status sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
7102025-04-30T06:42:38.500ZDEBGtest_snapshot_basic (dropshot): registered endpoint local_addr = [::1]:41923 method = GET path = /datasets sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
7112025-04-30T06:42:38.500ZDEBGtest_snapshot_basic (dropshot): registered endpoint local_addr = [::1]:41923 method = PUT path = /disks/{disk_id} sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
7122025-04-30T06:42:38.500ZDEBGtest_snapshot_basic (dropshot): registered endpoint local_addr = [::1]:41923 method = POST path = /disks/{disk_id}/poke sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
7132025-04-30T06:42:38.500ZDEBGtest_snapshot_basic (dropshot): registered endpoint local_addr = [::1]:41923 method = PUT path = /eip-gateways sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
7142025-04-30T06:42:38.500ZDEBGtest_snapshot_basic (dropshot): registered endpoint local_addr = [::1]:41923 method = GET path = /inventory sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
7152025-04-30T06:42:38.500ZDEBGtest_snapshot_basic (dropshot): registered endpoint local_addr = [::1]:41923 method = GET path = /network-bootstore-config sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
7162025-04-30T06:42:38.500ZDEBGtest_snapshot_basic (dropshot): registered endpoint local_addr = [::1]:41923 method = PUT path = /network-bootstore-config sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
7172025-04-30T06:42:38.500ZDEBGtest_snapshot_basic (dropshot): registered endpoint local_addr = [::1]:41923 method = PUT path = /omicron-config sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
7182025-04-30T06:42:38.500ZDEBGtest_snapshot_basic (dropshot): registered endpoint local_addr = [::1]:41923 method = GET path = /omicron-physical-disks sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
7192025-04-30T06:42:38.500ZDEBGtest_snapshot_basic (dropshot): registered endpoint local_addr = [::1]:41923 method = GET path = /sled-identifiers sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
7202025-04-30T06:42:38.500ZDEBGtest_snapshot_basic (dropshot): registered endpoint local_addr = [::1]:41923 method = GET path = /sled-role sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
7212025-04-30T06:42:38.500ZDEBGtest_snapshot_basic (dropshot): registered endpoint local_addr = [::1]:41923 method = PUT path = /sleds sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
7222025-04-30T06:42:38.500ZDEBGtest_snapshot_basic (dropshot): registered endpoint local_addr = [::1]:41923 method = GET path = /support/dladm-info sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
7232025-04-30T06:42:38.500ZDEBGtest_snapshot_basic (dropshot): registered endpoint local_addr = [::1]:41923 method = GET path = /support/ipadm-info sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
7242025-04-30T06:42:38.500ZDEBGtest_snapshot_basic (dropshot): registered endpoint local_addr = [::1]:41923 method = GET path = /support/logs/download/{zone} sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
7252025-04-30T06:42:38.500ZDEBGtest_snapshot_basic (dropshot): registered endpoint local_addr = [::1]:41923 method = GET path = /support/logs/zones sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
7262025-04-30T06:42:38.500ZDEBGtest_snapshot_basic (dropshot): registered endpoint local_addr = [::1]:41923 method = GET path = /support/nvmeadm-info sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
7272025-04-30T06:42:38.500ZDEBGtest_snapshot_basic (dropshot): registered endpoint local_addr = [::1]:41923 method = GET path = /support/pargs-info sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
7282025-04-30T06:42:38.500ZDEBGtest_snapshot_basic (dropshot): registered endpoint local_addr = [::1]:41923 method = GET path = /support/pfiles-info sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
7292025-04-30T06:42:38.500ZDEBGtest_snapshot_basic (dropshot): registered endpoint local_addr = [::1]:41923 method = GET path = /support/pstack-info sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
7302025-04-30T06:42:38.500ZDEBGtest_snapshot_basic (dropshot): registered endpoint local_addr = [::1]:41923 method = GET path = /support/zfs-info sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
7312025-04-30T06:42:38.500ZDEBGtest_snapshot_basic (dropshot): registered endpoint local_addr = [::1]:41923 method = GET path = /support/zoneadm-info sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
7322025-04-30T06:42:38.500ZDEBGtest_snapshot_basic (dropshot): registered endpoint local_addr = [::1]:41923 method = GET path = /support/zpool-info sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
7332025-04-30T06:42:38.500ZDEBGtest_snapshot_basic (dropshot): registered endpoint local_addr = [::1]:41923 method = GET path = /support-bundles/{zpool_id}/{dataset_id} sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
7342025-04-30T06:42:38.500ZDEBGtest_snapshot_basic (dropshot): registered endpoint local_addr = [::1]:41923 method = DELETE path = /support-bundles/{zpool_id}/{dataset_id}/{support_bundle_id} sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
7352025-04-30T06:42:38.500ZDEBGtest_snapshot_basic (dropshot): registered endpoint local_addr = [::1]:41923 method = POST path = /support-bundles/{zpool_id}/{dataset_id}/{support_bundle_id} sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
7362025-04-30T06:42:38.501ZDEBGtest_snapshot_basic (dropshot): registered endpoint local_addr = [::1]:41923 method = GET path = /support-bundles/{zpool_id}/{dataset_id}/{support_bundle_id}/download sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
7372025-04-30T06:42:38.501ZDEBGtest_snapshot_basic (dropshot): registered endpoint local_addr = [::1]:41923 method = HEAD path = /support-bundles/{zpool_id}/{dataset_id}/{support_bundle_id}/download sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
7382025-04-30T06:42:38.501ZDEBGtest_snapshot_basic (dropshot): registered endpoint local_addr = [::1]:41923 method = GET path = /support-bundles/{zpool_id}/{dataset_id}/{support_bundle_id}/download/{file} sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
7392025-04-30T06:42:38.501ZDEBGtest_snapshot_basic (dropshot): registered endpoint local_addr = [::1]:41923 method = HEAD path = /support-bundles/{zpool_id}/{dataset_id}/{support_bundle_id}/download/{file} sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
7402025-04-30T06:42:38.501ZDEBGtest_snapshot_basic (dropshot): registered endpoint local_addr = [::1]:41923 method = GET path = /support-bundles/{zpool_id}/{dataset_id}/{support_bundle_id}/index sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
7412025-04-30T06:42:38.501ZDEBGtest_snapshot_basic (dropshot): registered endpoint local_addr = [::1]:41923 method = HEAD path = /support-bundles/{zpool_id}/{dataset_id}/{support_bundle_id}/index sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
7422025-04-30T06:42:38.501ZDEBGtest_snapshot_basic (dropshot): registered endpoint local_addr = [::1]:41923 method = POST path = /switch-ports sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
7432025-04-30T06:42:38.501ZDEBGtest_snapshot_basic (dropshot): registered endpoint local_addr = [::1]:41923 method = GET path = /timesync sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
7442025-04-30T06:42:38.501ZDEBGtest_snapshot_basic (dropshot): registered endpoint local_addr = [::1]:41923 method = DELETE path = /v2p sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
7452025-04-30T06:42:38.501ZDEBGtest_snapshot_basic (dropshot): registered endpoint local_addr = [::1]:41923 method = GET path = /v2p sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
7462025-04-30T06:42:38.501ZDEBGtest_snapshot_basic (dropshot): registered endpoint local_addr = [::1]:41923 method = PUT path = /v2p sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
7472025-04-30T06:42:38.501ZDEBGtest_snapshot_basic (dropshot): registered endpoint local_addr = [::1]:41923 method = DELETE path = /vmms/{propolis_id} sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
7482025-04-30T06:42:38.501ZDEBGtest_snapshot_basic (dropshot): registered endpoint local_addr = [::1]:41923 method = PUT path = /vmms/{propolis_id} sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
7492025-04-30T06:42:38.501ZDEBGtest_snapshot_basic (dropshot): registered endpoint local_addr = [::1]:41923 method = POST path = /vmms/{propolis_id}/disks/{disk_id}/snapshot sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
7502025-04-30T06:42:38.501ZDEBGtest_snapshot_basic (dropshot): registered endpoint local_addr = [::1]:41923 method = DELETE path = /vmms/{propolis_id}/external-ip sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
7512025-04-30T06:42:38.501ZDEBGtest_snapshot_basic (dropshot): registered endpoint local_addr = [::1]:41923 method = PUT path = /vmms/{propolis_id}/external-ip sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
7522025-04-30T06:42:38.501ZDEBGtest_snapshot_basic (dropshot): registered endpoint local_addr = [::1]:41923 method = POST path = /vmms/{propolis_id}/poke sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
7532025-04-30T06:42:38.501ZDEBGtest_snapshot_basic (dropshot): registered endpoint local_addr = [::1]:41923 method = POST path = /vmms/{propolis_id}/poke-single-step sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
7542025-04-30T06:42:38.501ZDEBGtest_snapshot_basic (dropshot): registered endpoint local_addr = [::1]:41923 method = POST path = /vmms/{propolis_id}/sim-migration-source sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
7552025-04-30T06:42:38.501ZDEBGtest_snapshot_basic (dropshot): registered endpoint local_addr = [::1]:41923 method = GET path = /vmms/{propolis_id}/state sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
7562025-04-30T06:42:38.501ZDEBGtest_snapshot_basic (dropshot): registered endpoint local_addr = [::1]:41923 method = PUT path = /vmms/{propolis_id}/state sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
7572025-04-30T06:42:38.501ZDEBGtest_snapshot_basic (dropshot): registered endpoint local_addr = [::1]:41923 method = PUT path = /vpc/{vpc_id}/firewall/rules sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
7582025-04-30T06:42:38.501ZDEBGtest_snapshot_basic (dropshot): registered endpoint local_addr = [::1]:41923 method = GET path = /vpc-routes sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
7592025-04-30T06:42:38.501ZDEBGtest_snapshot_basic (dropshot): registered endpoint local_addr = [::1]:41923 method = PUT path = /vpc-routes sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
7602025-04-30T06:42:38.501ZDEBGtest_snapshot_basic (dropshot): registered endpoint local_addr = [::1]:41923 method = GET path = /zones sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
7612025-04-30T06:42:38.501ZDEBGtest_snapshot_basic (dropshot): registered endpoint local_addr = [::1]:41923 method = POST path = /zones/bundle-cleanup sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
7622025-04-30T06:42:38.501ZDEBGtest_snapshot_basic (dropshot): registered endpoint local_addr = [::1]:41923 method = GET path = /zones/bundle-cleanup/context sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
7632025-04-30T06:42:38.501ZDEBGtest_snapshot_basic (dropshot): registered endpoint local_addr = [::1]:41923 method = PUT path = /zones/bundle-cleanup/context sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
7642025-04-30T06:42:38.501ZDEBGtest_snapshot_basic (dropshot): registered endpoint local_addr = [::1]:41923 method = GET path = /zones/bundle-cleanup/utilization sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
7652025-04-30T06:42:38.501ZDEBGtest_snapshot_basic (dropshot): registered endpoint local_addr = [::1]:41923 method = GET path = /zones/bundles sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
7662025-04-30T06:42:38.501ZDEBGtest_snapshot_basic (dropshot): registered endpoint local_addr = [::1]:41923 method = GET path = /zones/bundles/{zone_name} sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
7672025-04-30T06:42:38.501ZDEBGtest_snapshot_basic (dropshot): registered endpoint local_addr = [::1]:41923 method = DELETE path = /zones/bundles/{zone_name}/{bundle_id} sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
7682025-04-30T06:42:38.501ZDEBGtest_snapshot_basic (dropshot): registered endpoint local_addr = [::1]:41923 method = GET path = /zones/bundles/{zone_name}/{bundle_id} sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
7692025-04-30T06:42:38.501ZDEBGtest_snapshot_basic (dropshot): registered endpoint local_addr = [::1]:41923 method = GET path = /zpools sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
7702025-04-30T06:42:38.501ZINFOtest_snapshot_basic (dropshot): listening local_addr = [::1]:41923 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
7712025-04-30T06:42:38.501ZDEBGtest_snapshot_basic (dropshot): successfully registered DTrace USDT probes local_addr = [::1]:41923 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
7722025-04-30T06:42:38.501ZDEBGtest_snapshot_basic (BackgroundTasks): activating background_task = v2p_manager iteration = 1 reason = Timeout
7732025-04-30T06:42:38.502ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database
7742025-04-30T06:42:38.502ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
7752025-04-30T06:42:38.503ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
7762025-04-30T06:42:38.503ZDEBGtest_snapshot_basic (BackgroundTasks): activating background_task = region_replacement iteration = 1 reason = Timeout
7772025-04-30T06:42:38.503ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database
7782025-04-30T06:42:38.503ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = region_replacement roles = RoleSet { roles: {} }
7792025-04-30T06:42:38.503ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database result = Ok(())
7802025-04-30T06:42:38.503ZDEBGtest_snapshot_basic (BackgroundTasks): activating background_task = region_replacement_driver iteration = 1 reason = Timeout
7812025-04-30T06:42:38.503ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database
7822025-04-30T06:42:38.503ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
7832025-04-30T06:42:38.504ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
7842025-04-30T06:42:38.504ZDEBGtest_snapshot_basic (BackgroundTasks): activating background_task = instance_watcher iteration = 1 reason = Timeout
7852025-04-30T06:42:38.504ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Fleet
7862025-04-30T06:42:38.504ZTRACtest_snapshot_basic (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = instance_watcher resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
7872025-04-30T06:42:38.504ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database
7882025-04-30T06:42:38.504ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
7892025-04-30T06:42:38.504ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
7902025-04-30T06:42:38.504ZDEBGtest_snapshot_basic (BackgroundTasks): activating background_task = instance_updater iteration = 1 reason = Timeout
7912025-04-30T06:42:38.504ZINFOtest_snapshot_basic (BackgroundTasks): background instance updater explicitly disabled background_task = instance_updater
7922025-04-30T06:42:38.505ZDEBGtest_snapshot_basic (BackgroundTasks): activation complete background_task = instance_updater elapsed = 303.735\u{b5}s iteration = 1 status = {"destroyed_active_vmms":0,"disabled":true,"failed_active_vmms":0,"query_errors":[],"saga_errors":[],"sagas_completed":0,"sagas_started":0,"terminated_active_migrations":0}
7932025-04-30T06:42:38.505ZDEBGtest_snapshot_basic (BackgroundTasks): activating background_task = instance_reincarnation iteration = 1 reason = Timeout
7942025-04-30T06:42:38.505ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_reincarnation resource = Database
7952025-04-30T06:42:38.505ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = instance_reincarnation roles = RoleSet { roles: {} }
7962025-04-30T06:42:38.506ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_reincarnation resource = Database result = Ok(())
7972025-04-30T06:42:38.506ZDEBGtest_snapshot_basic (BackgroundTasks): activating background_task = service_firewall_rule_propagation iteration = 1 reason = Timeout
7982025-04-30T06:42:38.506ZDEBGtest_snapshot_basic (service-firewall-rule-progator): starting background task for service firewall rule propagation background_task = service_firewall_rule_propagation
7992025-04-30T06:42:38.506ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database
8002025-04-30T06:42:38.506ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
8012025-04-30T06:42:38.506ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
8022025-04-30T06:42:38.507ZDEBGtest_snapshot_basic (BackgroundTasks): activating background_task = vpc_route_manager iteration = 1 reason = Timeout
8032025-04-30T06:42:38.507ZINFOtest_snapshot_basic (BackgroundTasks): VPC route manager running background_task = vpc_route_manager
8042025-04-30T06:42:38.507ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet
8052025-04-30T06:42:38.507ZTRACtest_snapshot_basic (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = vpc_route_manager resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
8062025-04-30T06:42:38.507ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database
8072025-04-30T06:42:38.507ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
8082025-04-30T06:42:38.507ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
8092025-04-30T06:42:38.507ZDEBGtest_snapshot_basic (BackgroundTasks): activating background_task = abandoned_vmm_reaper iteration = 1 reason = Timeout
8102025-04-30T06:42:38.507ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = abandoned_vmm_reaper resource = Database
8112025-04-30T06:42:38.507ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = abandoned_vmm_reaper roles = RoleSet { roles: {} }
8122025-04-30T06:42:38.508ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = abandoned_vmm_reaper resource = Database result = Ok(())
8132025-04-30T06:42:38.508ZDEBGtest_snapshot_basic (BackgroundTasks): activating background_task = saga_recovery iteration = 1 reason = Timeout
8142025-04-30T06:42:38.508ZDEBGtest_snapshot_basic (SagaRecoverer): listing candidate sagas for recovery
8152025-04-30T06:42:38.508ZTRACtest_snapshot_basic (SagaRecoverer): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000003a8a, .. }) resource = Database
8162025-04-30T06:42:38.508ZDEBGtest_snapshot_basic (SagaRecoverer): roles roles = RoleSet { roles: {} }
8172025-04-30T06:42:38.508ZDEBGtest_snapshot_basic (SagaRecoverer): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000003a8a, .. }) resource = Database result = Ok(())
8182025-04-30T06:42:38.508ZDEBGtest_snapshot_basic (BackgroundTasks): activating background_task = lookup_region_port iteration = 1 reason = Timeout
8192025-04-30T06:42:38.508ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = lookup_region_port resource = Database
8202025-04-30T06:42:38.508ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = lookup_region_port roles = RoleSet { roles: {} }
8212025-04-30T06:42:38.509ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = lookup_region_port resource = Database result = Ok(())
8222025-04-30T06:42:38.510ZDEBGtest_snapshot_basic (BackgroundTasks): activating background_task = region_snapshot_replacement_start iteration = 1 reason = Timeout
8232025-04-30T06:42:38.510ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_start resource = Database
8242025-04-30T06:42:38.510ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = region_snapshot_replacement_start roles = RoleSet { roles: {} }
8252025-04-30T06:42:38.510ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_start resource = Database result = Ok(())
8262025-04-30T06:42:38.511ZDEBGtest_snapshot_basic (BackgroundTasks): activating background_task = region_snapshot_replacement_garbage_collection iteration = 1 reason = Timeout
8272025-04-30T06:42:38.511ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_garbage_collection resource = Database
8282025-04-30T06:42:38.511ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = region_snapshot_replacement_garbage_collection roles = RoleSet { roles: {} }
8292025-04-30T06:42:38.511ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_garbage_collection resource = Database result = Ok(())
8302025-04-30T06:42:38.512ZDEBGtest_snapshot_basic (BackgroundTasks): activating background_task = region_snapshot_replacement_step iteration = 1 reason = Timeout
8312025-04-30T06:42:38.512ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database
8322025-04-30T06:42:38.512ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
8332025-04-30T06:42:38.514ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
8342025-04-30T06:42:38.514ZDEBGtest_snapshot_basic (BackgroundTasks): activating background_task = region_snapshot_replacement_finish iteration = 1 reason = Timeout
8352025-04-30T06:42:38.514ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_finish resource = Database
8362025-04-30T06:42:38.514ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = region_snapshot_replacement_finish roles = RoleSet { roles: {} }
8372025-04-30T06:42:38.514ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_finish resource = Database result = Ok(())
8382025-04-30T06:42:38.514ZDEBGtest_snapshot_basic (BackgroundTasks): activating background_task = tuf_artifact_replication iteration = 1 reason = Timeout
8392025-04-30T06:42:38.514ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = tuf_artifact_replication resource = Fleet
8402025-04-30T06:42:38.514ZTRACtest_snapshot_basic (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = tuf_artifact_replication resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
8412025-04-30T06:42:38.514ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = tuf_artifact_replication resource = Database
8422025-04-30T06:42:38.514ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = tuf_artifact_replication roles = RoleSet { roles: {} }
8432025-04-30T06:42:38.515ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = tuf_artifact_replication resource = Database result = Ok(())
8442025-04-30T06:42:38.515ZDEBGtest_snapshot_basic (BackgroundTasks): activating background_task = read_only_region_replacement_start iteration = 1 reason = Timeout
8452025-04-30T06:42:38.515ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = read_only_region_replacement_start resource = Database
8462025-04-30T06:42:38.515ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = read_only_region_replacement_start roles = RoleSet { roles: {} }
8472025-04-30T06:42:38.516ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = read_only_region_replacement_start resource = Database result = Ok(())
8482025-04-30T06:42:38.516ZDEBGtest_snapshot_basic (BackgroundTasks): activating background_task = webhook_dispatcher iteration = 1 reason = Timeout
8492025-04-30T06:42:38.516ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = webhook_dispatcher resource = Database
8502025-04-30T06:42:38.516ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = webhook_dispatcher roles = RoleSet { roles: {} }
8512025-04-30T06:42:38.517ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = webhook_dispatcher resource = Database result = Ok(())
8522025-04-30T06:42:38.517ZDEBGtest_snapshot_basic (BackgroundTasks): activating background_task = webhook_deliverator iteration = 1 reason = Timeout
8532025-04-30T06:42:38.517ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = webhook_deliverator resource = Database
8542025-04-30T06:42:38.517ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = webhook_deliverator roles = RoleSet { roles: {} }
8552025-04-30T06:42:38.517ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = webhook_deliverator resource = Database result = Ok(())
8562025-04-30T06:42:38.518ZDEBGtest_snapshot_basic (omicron_sled_agent::sim::Server): contacting server nexus sled_id = b6d65341-167c-41df-9b5c-41cded99c229
8572025-04-30T06:42:38.519ZDEBGtest_snapshot_basic (NexusClient): client request body = Some(Body) method = POST sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = http://127.0.0.1:61468/sled-agents/b6d65341-167c-41df-9b5c-41cded99c229
8582025-04-30T06:42:38.520ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = DnsConfig
8592025-04-30T06:42:38.520ZTRACtest_snapshot_basic (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = dns_config_internal dns_group = internal resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
8602025-04-30T06:42:38.520ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database
8612025-04-30T06:42:38.520ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
8622025-04-30T06:42:38.520ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
8632025-04-30T06:42:38.521ZINFOtest_snapshot_basic (dropshot_internal): accepted connection local_addr = 127.0.0.1:61468 remote_addr = 127.0.0.1:36310
8642025-04-30T06:42:38.522ZTRACtest_snapshot_basic (dns): buffer buffer = Length: 75 (0x4b) bytes\n0000: 4d 90 01 00 00 01 00 00 00 00 00 01 0c 5f 6e 61 M............_na\n0010: 6d 65 73 65 72 76 69 63 65 04 5f 74 63 70 0d 63 meservice._tcp.c\n0020: 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 ontrol-plane.oxi\n0030: 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 21 00 01 de.internal..!..\n0040: 00 00 29 04 d0 00 00 00 00 00 00 ..)........ kind = dns peer_addr = [::1]:61548 req_id = 173c0ac1-0a1e-461c-a099-88a8c1f44dff
8652025-04-30T06:42:38.523ZDEBGtest_snapshot_basic (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 19856,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_nameservice._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_nameservice._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 12,\n 95,\n 110,\n 97,\n 109,\n 101,\n 115,\n 101,\n 114,\n 118,\n 105,\n 99,\n 101,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:61548 req_id = 173c0ac1-0a1e-461c-a099-88a8c1f44dff
8662025-04-30T06:42:38.524ZERROtest_snapshot_basic (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 19856, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_nameservice._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_nameservice._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 12, 95, 110, 97, 109, 101, 115, 101, 114, 118, 105, 99, 101, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } SERVFAIL: server is not authoritative for name: "_nameservice._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:61548 req_id = 173c0ac1-0a1e-461c-a099-88a8c1f44dff
8672025-04-30T06:42:38.531ZTRACtest_snapshot_basic (dns): buffer buffer = Length: 76 (0x4c) bytes\n0000: ec 2d 01 00 00 01 00 00 00 00 00 01 0d 5f 65 78 .-..........._ex\n0010: 74 65 72 6e 61 6c 2d 64 6e 73 04 5f 74 63 70 0d ternal-dns._tcp.\n0020: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0030: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 21 00 ide.internal..!.\n0040: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:55671 req_id = 3d5b6702-c082-405c-bedc-acbd6d1e3744
8682025-04-30T06:42:38.531ZDEBGtest_snapshot_basic (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 60461,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_external-dns._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_external-dns._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 13,\n 95,\n 101,\n 120,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 45,\n 100,\n 110,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:55671 req_id = 3d5b6702-c082-405c-bedc-acbd6d1e3744
8692025-04-30T06:42:38.532ZERROtest_snapshot_basic (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 60461, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_external-dns._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_external-dns._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 13, 95, 101, 120, 116, 101, 114, 110, 97, 108, 45, 100, 110, 115, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } SERVFAIL: server is not authoritative for name: "_external-dns._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:55671 req_id = 3d5b6702-c082-405c-bedc-acbd6d1e3744
8702025-04-30T06:42:38.534ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = DnsConfig
8712025-04-30T06:42:38.534ZTRACtest_snapshot_basic (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = dns_config_external dns_group = external resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
8722025-04-30T06:42:38.534ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database
8732025-04-30T06:42:38.534ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
8742025-04-30T06:42:38.535ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
8752025-04-30T06:42:38.538ZWARNtest_snapshot_basic (BackgroundTasks): failed to lookup DNS servers background_task = dns_servers_internal dns_group = internal error = no record found for Query { name: Name("_nameservice._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN }
8762025-04-30T06:42:38.538ZDEBGtest_snapshot_basic (BackgroundTasks): activation complete background_task = dns_servers_internal dns_group = internal elapsed = 122.621886ms iteration = 1 status = {"error":"failed to read list of DNS servers: no record found for Query { name: Name(\\"_nameservice._tcp.control-plane.oxide.internal.\\"), query_type: SRV, query_class: IN }"}
8772025-04-30T06:42:38.538ZWARNtest_snapshot_basic (BackgroundTasks): failed to lookup DNS servers background_task = dns_servers_external dns_group = external error = no record found for Query { name: Name("_external-dns._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN }
8782025-04-30T06:42:38.539ZDEBGtest_snapshot_basic (BackgroundTasks): activation complete background_task = dns_servers_external dns_group = external elapsed = 119.884873ms iteration = 1 status = {"error":"failed to read list of DNS servers: no record found for Query { name: Name(\\"_external-dns._tcp.control-plane.oxide.internal.\\"), query_type: SRV, query_class: IN }"}
8792025-04-30T06:42:38.540ZTRACtest_snapshot_basic (dropshot_internal): incoming request local_addr = 127.0.0.1:61468 method = POST remote_addr = 127.0.0.1:36310 req_id = b266969b-23a8-43a5-8e4f-bbd8ee49a8ca uri = /sled-agents/b6d65341-167c-41df-9b5c-41cded99c229
8802025-04-30T06:42:38.543ZINFOtest_snapshot_basic (nexus): registered sled agent sled_uuid = b6d65341-167c-41df-9b5c-41cded99c229
8812025-04-30T06:42:38.558ZINFOtest_snapshot_basic (BackgroundTasks): Metric producer GC complete (no errors) background_task = metrics_producer_gc expiration = 2025-04-30 06:32:38.419484759 UTC pruned = {}
8822025-04-30T06:42:38.558ZDEBGtest_snapshot_basic (BackgroundTasks): activation complete background_task = metrics_producer_gc elapsed = 138.899087ms iteration = 1 status = {"expiration":"2025-04-30T06:32:38.419484759Z","pruned":[]}
8832025-04-30T06:42:38.558ZTRACtest_snapshot_basic (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal
8842025-04-30T06:42:38.567ZTRACtest_snapshot_basic (dns): buffer buffer = Length: 72 (0x48) bytes\n0000: cf a4 01 00 00 01 00 00 00 00 00 01 09 5f 64 65 ............._de\n0010: 6e 64 72 69 74 65 04 5f 74 63 70 0d 63 6f 6e 74 ndrite._tcp.cont\n0020: 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 rol-plane.oxide.\n0030: 69 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 internal..!....)\n0040: 04 d0 00 00 00 00 00 00 ........ kind = dns peer_addr = [::1]:60912 req_id = 9e6192b2-0bb4-465c-9b1b-c159fcc17fac
8852025-04-30T06:42:38.567ZDEBGtest_snapshot_basic (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 53156,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_dendrite._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_dendrite._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 9,\n 95,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:60912 req_id = 9e6192b2-0bb4-465c-9b1b-c159fcc17fac
8862025-04-30T06:42:38.568ZERROtest_snapshot_basic (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 53156, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_dendrite._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 9, 95, 100, 101, 110, 100, 114, 105, 116, 101, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } SERVFAIL: server is not authoritative for name: "_dendrite._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:60912 req_id = 9e6192b2-0bb4-465c-9b1b-c159fcc17fac
8872025-04-30T06:42:38.569ZTRACtest_snapshot_basic (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal
8882025-04-30T06:42:38.571ZERROtest_snapshot_basic (BackgroundTasks): failed to resolve addresses for Dendrite services background_task = nat_v4_garbage_collector error = no record found for Query { name: Name("_dendrite._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN }
8892025-04-30T06:42:38.571ZERROtest_snapshot_basic (BackgroundTasks): failed to resolve addresses for Dendrite services background_task = nat_v4_garbage_collector error = no record found for Query { name: Name("_dendrite._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN }
8902025-04-30T06:42:38.571ZDEBGtest_snapshot_basic (BackgroundTasks): activation complete background_task = nat_v4_garbage_collector elapsed = 149.758765ms iteration = 1 status = {"error":"failed to resolve addresses for Dendrite services: no record found for Query { name: Name(\\"_dendrite._tcp.control-plane.oxide.internal.\\"), query_type: SRV, query_class: IN }"}
8912025-04-30T06:42:38.606ZTRACtest_snapshot_basic (dns): buffer buffer = Length: 72 (0x48) bytes\n0000: eb c2 01 00 00 01 00 00 00 00 00 01 09 5f 64 65 ............._de\n0010: 6e 64 72 69 74 65 04 5f 74 63 70 0d 63 6f 6e 74 ndrite._tcp.cont\n0020: 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 rol-plane.oxide.\n0030: 69 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 internal..!....)\n0040: 04 d0 00 00 00 00 00 00 ........ kind = dns peer_addr = [::1]:56463 req_id = c0bb0282-6ba6-4335-b219-56c376955778
8922025-04-30T06:42:38.611ZDEBGtest_snapshot_basic (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 60354,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_dendrite._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_dendrite._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 9,\n 95,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:56463 req_id = c0bb0282-6ba6-4335-b219-56c376955778
8932025-04-30T06:42:38.612ZERROtest_snapshot_basic (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 60354, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_dendrite._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 9, 95, 100, 101, 110, 100, 114, 105, 116, 101, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } SERVFAIL: server is not authoritative for name: "_dendrite._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:56463 req_id = c0bb0282-6ba6-4335-b219-56c376955778
8942025-04-30T06:42:38.614ZDEBGtest_snapshot_basic (BackgroundTasks): activation complete background_task = phantom_disks elapsed = 190.872361ms iteration = 1 status = {"phantom_disk_deleted_err":0,"phantom_disk_deleted_ok":0}
8952025-04-30T06:42:38.617ZERROtest_snapshot_basic (BackgroundTasks): failed to resolve addresses for Dendrite services background_task = bfd_manager error = no record found for Query { name: Name("_dendrite._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN }
8962025-04-30T06:42:38.617ZERROtest_snapshot_basic (BackgroundTasks): failed to resolve addresses for Dendrite services background_task = bfd_manager error = no record found for Query { name: Name("_dendrite._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN }
8972025-04-30T06:42:38.617ZDEBGtest_snapshot_basic (BackgroundTasks): activation complete background_task = bfd_manager elapsed = 195.463999ms iteration = 1 status = {"error":"failed to resolve addresses for Dendrite services: no record found for Query { name: Name(\\"_dendrite._tcp.control-plane.oxide.internal.\\"), query_type: SRV, query_class: IN }"}
8982025-04-30T06:42:38.618ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
8992025-04-30T06:42:38.623ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Fleet result = Ok(())
9002025-04-30T06:42:38.623ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database
9012025-04-30T06:42:38.624ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
9022025-04-30T06:42:38.624ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
9032025-04-30T06:42:38.686ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
9042025-04-30T06:42:38.700ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
9052025-04-30T06:42:38.700ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database
9062025-04-30T06:42:38.700ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
9072025-04-30T06:42:38.700ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
9082025-04-30T06:42:38.700ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
9092025-04-30T06:42:38.704ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
9102025-04-30T06:42:38.705ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
9112025-04-30T06:42:38.705ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
9122025-04-30T06:42:38.705ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
9132025-04-30T06:42:38.706ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
9142025-04-30T06:42:38.711ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Fleet result = Ok(())
9152025-04-30T06:42:38.711ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database
9162025-04-30T06:42:38.711ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
9172025-04-30T06:42:38.712ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
9182025-04-30T06:42:38.712ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
9192025-04-30T06:42:38.722ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Inventory result = Ok(())
9202025-04-30T06:42:38.722ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database
9212025-04-30T06:42:38.722ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
9222025-04-30T06:42:38.723ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
9232025-04-30T06:42:38.723ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
9242025-04-30T06:42:38.732ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Fleet result = Ok(())
9252025-04-30T06:42:38.732ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database
9262025-04-30T06:42:38.732ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
9272025-04-30T06:42:38.733ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
9282025-04-30T06:42:38.736ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database
9292025-04-30T06:42:38.736ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
9302025-04-30T06:42:38.738ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
9312025-04-30T06:42:38.804ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database
9322025-04-30T06:42:38.805ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
9332025-04-30T06:42:38.805ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
9342025-04-30T06:42:38.820ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database
9352025-04-30T06:42:38.821ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = region_replacement roles = RoleSet { roles: {} }
9362025-04-30T06:42:38.821ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database result = Ok(())
9372025-04-30T06:42:38.873ZDEBGtest_snapshot_basic (BackgroundTasks): activation complete background_task = lookup_region_port elapsed = 364.502237ms iteration = 1 status = {"errors":[],"found_port_ok":[]}
9382025-04-30T06:42:38.993ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
9392025-04-30T06:42:38.999ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
9402025-04-30T06:42:39.000ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet
9412025-04-30T06:42:39.000ZTRACtest_snapshot_basic (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = vpc_route_manager resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
9422025-04-30T06:42:39.000ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database
9432025-04-30T06:42:39.000ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
9442025-04-30T06:42:39.004ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
9452025-04-30T06:42:39.004ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
9462025-04-30T06:42:39.010ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Fleet result = Ok(())
9472025-04-30T06:42:39.010ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database
9482025-04-30T06:42:39.010ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
9492025-04-30T06:42:39.010ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
9502025-04-30T06:42:39.041ZINFOtest_snapshot_basic (SagaRecoverer): listed in-progress sagas count = 0
9512025-04-30T06:42:39.054ZDEBGtest_snapshot_basic (BackgroundTasks): activation complete background_task = saga_recovery elapsed = 545.695613ms iteration = 1 status = {"last_pass":{"Success":{"nfailed":0,"nfound":0,"nrecovered":0,"nremoved":0,"nskipped":0}},"ntotal_failures":0,"ntotal_finished":0,"ntotal_recovered":0,"ntotal_sec_errors_bad_state":0,"ntotal_sec_errors_missing":0,"ntotal_started":0,"recent_failures":[],"recent_recoveries":[]}
9522025-04-30T06:42:39.084ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_start resource = Database
9532025-04-30T06:42:39.091ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = region_snapshot_replacement_start roles = RoleSet { roles: {} }
9542025-04-30T06:42:39.092ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_start resource = Database result = Ok(())
9552025-04-30T06:42:39.093ZTRACtest_snapshot_basic (BackgroundTasks): no more instances in need of reincarnation background_task = instance_reincarnation reincarnation_reason = instance failed total_found = 0
9562025-04-30T06:42:39.093ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_reincarnation resource = Database
9572025-04-30T06:42:39.093ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = instance_reincarnation roles = RoleSet { roles: {} }
9582025-04-30T06:42:39.094ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_reincarnation resource = Database result = Ok(())
9592025-04-30T06:42:39.098ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database
9602025-04-30T06:42:39.098ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
9612025-04-30T06:42:39.099ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
9622025-04-30T06:42:39.114ZDEBGtest_snapshot_basic (BackgroundTasks): activation complete background_task = region_snapshot_replacement_finish elapsed = 600.236933ms iteration = 1 status = {"errors":[],"finish_invoked_ok":[]}
9632025-04-30T06:42:39.114ZDEBGtest_snapshot_basic (BackgroundTasks): Found abandoned VMMs background_task = abandoned_vmm_reaper count = 0 total = 0
9642025-04-30T06:42:39.114ZINFOtest_snapshot_basic (BackgroundTasks): Abandoned VMMs reaped background_task = abandoned_vmm_reaper sled_reservations_deleted = 0 vmms_already_deleted = 0 vmms_deleted = 0 vmms_found = 0
9652025-04-30T06:42:39.114ZDEBGtest_snapshot_basic (BackgroundTasks): activation complete background_task = abandoned_vmm_reaper elapsed = 606.978516ms iteration = 1 status = {"errors":[],"sled_reservations_deleted":0,"vmms_already_deleted":0,"vmms_deleted":0,"vmms_found":0}
9662025-04-30T06:42:39.116ZDEBGtest_snapshot_basic (BackgroundTasks): activation complete background_task = region_snapshot_replacement_garbage_collection elapsed = 605.62578ms iteration = 1 status = {"errors":[],"garbage_collect_requested":[]}
9672025-04-30T06:42:39.117ZDEBGtest_snapshot_basic (BackgroundTasks): activation complete background_task = read_only_region_replacement_start elapsed = 601.554809ms iteration = 1 status = {"errors":[],"requests_created_ok":[]}
9682025-04-30T06:42:39.117ZDEBGtest_snapshot_basic (BackgroundTasks): activation complete background_task = webhook_deliverator elapsed = 600.623774ms iteration = 1 status = {"by_rx":{},"error":null}
9692025-04-30T06:42:39.117ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = tuf_artifact_replication roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
9702025-04-30T06:42:39.122ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = tuf_artifact_replication resource = Fleet result = Ok(())
9712025-04-30T06:42:39.122ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = tuf_artifact_replication resource = Fleet
9722025-04-30T06:42:39.122ZTRACtest_snapshot_basic (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = tuf_artifact_replication resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
9732025-04-30T06:42:39.122ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = tuf_artifact_replication resource = Database
9742025-04-30T06:42:39.122ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = tuf_artifact_replication roles = RoleSet { roles: {} }
9752025-04-30T06:42:39.122ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = tuf_artifact_replication resource = Database result = Ok(())
9762025-04-30T06:42:39.129ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database
9772025-04-30T06:42:39.129ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
9782025-04-30T06:42:39.130ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
9792025-04-30T06:42:39.130ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
9802025-04-30T06:42:39.137ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = DnsConfig result = Ok(())
9812025-04-30T06:42:39.150ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
9822025-04-30T06:42:39.155ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = DnsConfig result = Ok(())
9832025-04-30T06:42:39.210ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = DnsConfig
9842025-04-30T06:42:39.210ZTRACtest_snapshot_basic (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = external_endpoints resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
9852025-04-30T06:42:39.211ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database
9862025-04-30T06:42:39.211ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
9872025-04-30T06:42:39.211ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
9882025-04-30T06:42:39.273ZWARNtest_snapshot_basic (BackgroundTasks): failed to read target blueprint background_task = blueprint_loader error = Internal Error: no target blueprint set
9892025-04-30T06:42:39.273ZDEBGtest_snapshot_basic (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 850.513623ms iteration = 1 status = {"error":"failed to read target blueprint: Internal Error: no target blueprint set"}
9902025-04-30T06:42:39.298ZWARNtest_snapshot_basic (BackgroundTasks): failed to read DNS config background_task = dns_config_internal dns_group = internal error = Internal Error: expected exactly one latest version for DNS group Internal, found 0
9912025-04-30T06:42:39.298ZDEBGtest_snapshot_basic (BackgroundTasks): activation complete background_task = dns_config_internal dns_group = internal elapsed = 885.079414ms iteration = 1 status = {"error":"failed to read DNS config: Internal Error: expected exactly one latest version for DNS group Internal, found 0"}
9922025-04-30T06:42:39.321ZDEBGtest_snapshot_basic (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = []
9932025-04-30T06:42:39.321ZTRACtest_snapshot_basic (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
9942025-04-30T06:42:39.322ZWARNtest_snapshot_basic (BackgroundTasks): failed to read DNS config background_task = dns_config_external dns_group = external error = Internal Error: expected exactly one latest version for DNS group External, found 0
9952025-04-30T06:42:39.322ZDEBGtest_snapshot_basic (BackgroundTasks): activation complete background_task = dns_config_external dns_group = external elapsed = 905.427997ms iteration = 1 status = {"error":"failed to read DNS config: Internal Error: expected exactly one latest version for DNS group External, found 0"}
9962025-04-30T06:42:39.322ZTRACtest_snapshot_basic (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: 0d db 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 ............._mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:54187 req_id = 00d44682-9c8b-4580-af6e-ba7fc4ddfed2
9972025-04-30T06:42:39.322ZDEBGtest_snapshot_basic (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 3547,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:54187 req_id = 00d44682-9c8b-4580-af6e-ba7fc4ddfed2
9982025-04-30T06:42:39.323ZERROtest_snapshot_basic (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 3547, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_mgs._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 4, 95, 109, 103, 115, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } SERVFAIL: server is not authoritative for name: "_mgs._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:54187 req_id = 00d44682-9c8b-4580-af6e-ba7fc4ddfed2
9992025-04-30T06:42:39.323ZWARNtest_snapshot_basic (BackgroundTasks): inventory collection failed background_task = inventory_collection error = failed to collect inventory: looking up MGS addresses: no record found for Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN }
10002025-04-30T06:42:39.323ZDEBGtest_snapshot_basic (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 898.952464ms iteration = 1 status = {"error":"failed to collect inventory: looking up MGS addresses: no record found for Query { name: Name(\\"_mgs._tcp.control-plane.oxide.internal.\\"), query_type: SRV, query_class: IN }"}
10012025-04-30T06:42:39.329ZWARNtest_snapshot_basic (BackgroundTasks): inventory collection is None background_task = service_zone_nat_tracker
10022025-04-30T06:42:39.329ZDEBGtest_snapshot_basic (BackgroundTasks): activation complete background_task = service_zone_nat_tracker elapsed = 902.491896ms iteration = 1 status = {"error":"inventory collection is None"}
10032025-04-30T06:42:39.341ZDEBGtest_snapshot_basic (BackgroundTasks): activation complete background_task = switch_port_config_manager elapsed = 912.905816ms iteration = 1 status = {}
10042025-04-30T06:42:39.380ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database
10052025-04-30T06:42:39.380ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
10062025-04-30T06:42:39.381ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
10072025-04-30T06:42:39.394ZDEBGtest_snapshot_basic (BackgroundTasks): activating background_task = inventory_collection iteration = 2 reason = Signaled
10082025-04-30T06:42:39.394ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
10092025-04-30T06:42:39.394ZTRACtest_snapshot_basic (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
10102025-04-30T06:42:39.394ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
10112025-04-30T06:42:39.394ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
10122025-04-30T06:42:39.395ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
10132025-04-30T06:42:39.395ZINFOtest_snapshot_basic (dropshot_internal): request completed latency_us = 855153 local_addr = 127.0.0.1:61468 method = POST remote_addr = 127.0.0.1:36310 req_id = b266969b-23a8-43a5-8e4f-bbd8ee49a8ca response_code = 204 uri = /sled-agents/b6d65341-167c-41df-9b5c-41cded99c229
10142025-04-30T06:42:39.395ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
10152025-04-30T06:42:39.399ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
10162025-04-30T06:42:39.399ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database
10172025-04-30T06:42:39.399ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
10182025-04-30T06:42:39.399ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
10192025-04-30T06:42:39.399ZTRACtest_snapshot_basic (BackgroundTasks): reprocessing 0 outdated webhook globs... background_task = webhook_dispatcher
10202025-04-30T06:42:39.399ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = webhook_dispatcher resource = Database
10212025-04-30T06:42:39.399ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = webhook_dispatcher roles = RoleSet { roles: {} }
10222025-04-30T06:42:39.400ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = webhook_dispatcher resource = Database result = Ok(())
10232025-04-30T06:42:39.400ZDEBGtest_snapshot_basic (NexusClient): client response result = Ok(Response { url: "http://127.0.0.1:61468/sled-agents/b6d65341-167c-41df-9b5c-41cded99c229", status: 204, headers: {"x-request-id": "b266969b-23a8-43a5-8e4f-bbd8ee49a8ca", "date": "Wed, 30 Apr 2025 06:42:38 GMT"} }) sled_id = b6d65341-167c-41df-9b5c-41cded99c229
10242025-04-30T06:42:39.400ZDEBGtest_snapshot_basic (init_with_steps): Running step start_crucible_pantry
10252025-04-30T06:42:39.402ZDEBGtest_snapshot_basic (dropshot): registered endpoint kind = pantry local_addr = [::1]:62733 method = GET path = /crucible/pantry/0 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
10262025-04-30T06:42:39.402ZDEBGtest_snapshot_basic (dropshot): registered endpoint kind = pantry local_addr = [::1]:62733 method = GET path = /crucible/pantry/0/job/{id}/is-finished sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
10272025-04-30T06:42:39.402ZDEBGtest_snapshot_basic (dropshot): registered endpoint kind = pantry local_addr = [::1]:62733 method = GET path = /crucible/pantry/0/job/{id}/ok sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
10282025-04-30T06:42:39.402ZDEBGtest_snapshot_basic (dropshot): registered endpoint kind = pantry local_addr = [::1]:62733 method = DELETE path = /crucible/pantry/0/volume/{id} sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
10292025-04-30T06:42:39.402ZDEBGtest_snapshot_basic (dropshot): registered endpoint kind = pantry local_addr = [::1]:62733 method = GET path = /crucible/pantry/0/volume/{id} sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
10302025-04-30T06:42:39.402ZDEBGtest_snapshot_basic (dropshot): registered endpoint kind = pantry local_addr = [::1]:62733 method = POST path = /crucible/pantry/0/volume/{id} sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
10312025-04-30T06:42:39.402ZDEBGtest_snapshot_basic (dropshot): registered endpoint kind = pantry local_addr = [::1]:62733 method = POST path = /crucible/pantry/0/volume/{id}/background sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
10322025-04-30T06:42:39.402ZDEBGtest_snapshot_basic (dropshot): registered endpoint kind = pantry local_addr = [::1]:62733 method = POST path = /crucible/pantry/0/volume/{id}/bulk-write sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
10332025-04-30T06:42:39.402ZDEBGtest_snapshot_basic (dropshot): registered endpoint kind = pantry local_addr = [::1]:62733 method = POST path = /crucible/pantry/0/volume/{id}/import-from-url sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
10342025-04-30T06:42:39.402ZDEBGtest_snapshot_basic (dropshot): registered endpoint kind = pantry local_addr = [::1]:62733 method = POST path = /crucible/pantry/0/volume/{id}/scrub sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
10352025-04-30T06:42:39.402ZDEBGtest_snapshot_basic (dropshot): registered endpoint kind = pantry local_addr = [::1]:62733 method = POST path = /crucible/pantry/0/volume/{id}/snapshot sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
10362025-04-30T06:42:39.402ZINFOtest_snapshot_basic (dropshot): listening kind = pantry local_addr = [::1]:62733 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
10372025-04-30T06:42:39.403ZDEBGtest_snapshot_basic (dropshot): successfully registered DTrace USDT probes kind = pantry local_addr = [::1]:62733 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
10382025-04-30T06:42:39.403ZINFOtest_snapshot_basic (omicron_sled_agent::sim::Server): Started Simulated Crucible Pantry address = [::1]:62733 kind = pantry sled_id = b6d65341-167c-41df-9b5c-41cded99c229
10392025-04-30T06:42:39.403ZDEBGtest_snapshot_basic (init_with_steps): Running step populate_internal_dns
10402025-04-30T06:42:39.403ZDEBGtest_snapshot_basic: Populating Internal DNS
10412025-04-30T06:42:39.495ZINFOtest_snapshot_basic: DNS population: DnsConfigParams { generation: Generation( 1, ), time_created: 2025-04-30T06:42:39.495031476Z, zones: [ DnsConfigZone { zone_name: "control-plane.oxide.internal", records: { "6508a503-8e09-46a1-8cbe-d797e5d9a3aa.host": [ Aaaa( ::1, ), ], "dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host": [ Aaaa( ::1, ), ], "_dendrite._tcp": [ Srv( Srv { prio: 0, weight: 0, port: 32976, target: "dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal", }, ), ], "_mgd._tcp": [ Srv( Srv { prio: 0, weight: 0, port: 41136, target: "dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal", }, ), ], "7e244b61-e4d7-4237-8aaa-92889da3453c.host": [ Aaaa( ::1, ), ], "bc9e0b06-9302-4b86-9e68-9a19d64ee75c.host": [ Aaaa( ::1, ), ], "_repo-depot._tcp": [ Srv( Srv { prio: 0, weight: 0, port: 46799, target: "b6d65341-167c-41df-9b5c-41cded99c229.sled.control-plane.oxide.internal", }, ), ], "_oximeter-reader._tcp": [ Srv( Srv { prio: 0, weight: 0, port: 9000, target: "1a03c89c-cd2f-45e9-854b-34d13d044c99.host.control-plane.oxide.internal", }, ), ], "_cockroach._tcp": [ Srv( Srv { prio: 0, weight: 0, port: 43851, target: "bc9e0b06-9302-4b86-9e68-9a19d64ee75c.host.control-plane.oxide.internal", }, ), ], "_crucible-pantry._tcp": [ Srv( Srv { prio: 0, weight: 0, port: 62733, target: "92ab293e-632a-4f12-897f-edcc61e0fc3b.host.control-plane.oxide.internal", }, ), ], "_nexus._tcp": [ Srv( Srv { prio: 0, weight: 0, port: 61468, target: "354a52d4-3078-4706-81ce-12c32f6bbeaa.host.control-plane.oxide.internal", }, ), ], "_clickhouse-admin-single-server._tcp": [ Srv( Srv { prio: 0, weight: 0, port: 8888, target: "1a03c89c-cd2f-45e9-854b-34d13d044c99.host.control-plane.oxide.internal", }, ), ], "_nameservice._tcp": [ Srv( Srv { prio: 0, weight: 0, port: 48482, target: "6508a503-8e09-46a1-8cbe-d797e5d9a3aa.host.control-plane.oxide.internal", }, ), ], "_external-dns._tcp": [ Srv( Srv { prio: 0, weight: 0, port: 46673, target: "7e244b61-e4d7-4237-8aaa-92889da3453c.host.control-plane.oxide.internal", }, ), ], "_mgs._tcp": [ Srv( Srv { prio: 0, weight: 0, port: 41502, target: "dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal", }, ), ], "1a03c89c-cd2f-45e9-854b-34d13d044c99.host": [ Aaaa( ::1, ), ], "354a52d4-3078-4706-81ce-12c32f6bbeaa.host": [ Aaaa( ::ffff:127.0.0.1, ), ], "_clickhouse._tcp": [ Srv( Srv { prio: 0, weight: 0, port: 52745, target: "1a03c89c-cd2f-45e9-854b-34d13d044c99.host.control-plane.oxide.internal", }, ), ], "_clickhouse-native._tcp": [ Srv( Srv { prio: 0, weight: 0, port: 9000, target: "1a03c89c-cd2f-45e9-854b-34d13d044c99.host.control-plane.oxide.internal", }, ), ], "b6d65341-167c-41df-9b5c-41cded99c229.sled": [ Aaaa( ::1, ), ], "92ab293e-632a-4f12-897f-edcc61e0fc3b.host": [ Aaaa( ::1, ), ], }, }, ], }
10422025-04-30T06:42:39.495ZDEBGtest_snapshot_basic: client request body = Some(Body) method = PUT uri = http://[::1]:48482/config
10432025-04-30T06:42:39.496ZINFOtest_snapshot_basic (BackgroundTasks): all instance checks complete background_task = instance_watcher pruned_instances = 0 total_completed = 0 total_failed = 0 total_incomplete = 0 total_instances = 0 update_sagas_queued = 0
10442025-04-30T06:42:39.496ZDEBGtest_snapshot_basic (BackgroundTasks): activation complete background_task = instance_watcher elapsed = 992.250966ms iteration = 1 status = {"failed_checks":{},"incomplete_checks":{},"instance_states":{},"pruned_instances":0,"total_instances":0,"update_sagas_queued":0}
10452025-04-30T06:42:39.497ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Fleet
10462025-04-30T06:42:39.497ZTRACtest_snapshot_basic (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = support_bundle_collector resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
10472025-04-30T06:42:39.497ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database
10482025-04-30T06:42:39.497ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
10492025-04-30T06:42:39.498ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
10502025-04-30T06:42:39.499ZDEBGtest_snapshot_basic (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 996.094083ms iteration = 1 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
10512025-04-30T06:42:39.500ZDEBGtest_snapshot_basic (BackgroundTasks): activation complete background_task = region_replacement elapsed = 996.931532ms iteration = 1 status = {"errors":[],"requests_completed_ok":[],"requests_created_ok":[],"start_invoked_ok":[]}
10522025-04-30T06:42:39.502ZINFOtest_snapshot_basic (http): accepted connection kind = dns local_addr = [::1]:48482 remote_addr = [::1]:49306
10532025-04-30T06:42:39.505ZTRACtest_snapshot_basic (http): incoming request kind = dns local_addr = [::1]:48482 method = PUT remote_addr = [::1]:49306 req_id = c6f164e7-e81c-47c5-89b7-1c3473dcd0bd uri = /config
10542025-04-30T06:42:39.507ZINFOtest_snapshot_basic (store): attempting generation update new_generation = 1 req_id = c6f164e7-e81c-47c5-89b7-1c3473dcd0bd
10552025-04-30T06:42:39.507ZINFOtest_snapshot_basic (store): pruning trees for generations newer than 0
10562025-04-30T06:42:39.507ZDEBGtest_snapshot_basic (store): creating tree tree_name = generation_1_zone_control-plane.oxide.internal
10572025-04-30T06:42:39.520ZDEBGtest_snapshot_basic (store): updating current config
10582025-04-30T06:42:39.521ZDEBGtest_snapshot_basic (store): flushing default tree
10592025-04-30T06:42:39.521ZINFOtest_snapshot_basic (store): pruning trees for generations older than 1 keep = 3
10602025-04-30T06:42:39.521ZINFOtest_snapshot_basic (store): updated generation new_generation = 1 req_id = c6f164e7-e81c-47c5-89b7-1c3473dcd0bd
10612025-04-30T06:42:39.521ZINFOtest_snapshot_basic (http): request completed kind = dns latency_us = 16798 local_addr = [::1]:48482 method = PUT remote_addr = [::1]:49306 req_id = c6f164e7-e81c-47c5-89b7-1c3473dcd0bd response_code = 204 uri = /config
10622025-04-30T06:42:39.524ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = tuf_artifact_replication roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
10632025-04-30T06:42:39.535ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = tuf_artifact_replication resource = Fleet result = Ok(())
10642025-04-30T06:42:39.535ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = tuf_artifact_replication resource = Database
10652025-04-30T06:42:39.535ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = tuf_artifact_replication roles = RoleSet { roles: {} }
10662025-04-30T06:42:39.536ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = tuf_artifact_replication resource = Database result = Ok(())
10672025-04-30T06:42:39.538ZDEBGtest_snapshot_basic: client response result = Ok(Response { url: "http://[::1]:48482/config", status: 204, headers: {"x-request-id": "c6f164e7-e81c-47c5-89b7-1c3473dcd0bd", "date": "Wed, 30 Apr 2025 06:42:38 GMT"} })
10682025-04-30T06:42:39.539ZDEBGtest_snapshot_basic (init_with_steps): Running step configure_sled_agents
10692025-04-30T06:42:39.685ZDEBGtest_snapshot_basic: client request body = Some(Body) method = PUT uri = http://[::1]:41923/omicron-config
10702025-04-30T06:42:39.687ZDEBGtest_snapshot_basic (BackgroundTasks): activation complete background_task = region_snapshot_replacement_start elapsed = 1.177713588s iteration = 1 status = {"errors":[],"requests_completed_ok":[],"requests_created_ok":[],"start_invoked_ok":[]}
10712025-04-30T06:42:39.687ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database
10722025-04-30T06:42:39.687ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
10732025-04-30T06:42:39.689ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
10742025-04-30T06:42:39.690ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
10752025-04-30T06:42:39.702ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = DnsConfig result = Ok(())
10762025-04-30T06:42:39.702ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database
10772025-04-30T06:42:39.702ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
10782025-04-30T06:42:39.703ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
10792025-04-30T06:42:39.703ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ById(001de000-074c-4000-8000-000000000000) }
10802025-04-30T06:42:39.703ZTRACtest_snapshot_basic (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = service_firewall_rule_propagation resource_id = 001de000-4401-4000-8000-000000000000 resource_type = Project
10812025-04-30T06:42:39.703ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database
10822025-04-30T06:42:39.703ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
10832025-04-30T06:42:39.703ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
10842025-04-30T06:42:39.708ZINFOtest_snapshot_basic (dropshot): accepted connection local_addr = [::1]:41923 remote_addr = [::1]:34449 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
10852025-04-30T06:42:39.710ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
10862025-04-30T06:42:39.716ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
10872025-04-30T06:42:39.717ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
10882025-04-30T06:42:39.717ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
10892025-04-30T06:42:39.717ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
10902025-04-30T06:42:39.731ZTRACtest_snapshot_basic (dropshot): incoming request local_addr = [::1]:41923 method = PUT remote_addr = [::1]:34449 req_id = 373aba86-635c-4c3d-8c40-ea2db8aa2553 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /omicron-config
10912025-04-30T06:42:39.743ZINFOtest_snapshot_basic (dropshot): request completed latency_us = 11425 local_addr = [::1]:41923 method = PUT remote_addr = [::1]:34449 req_id = 373aba86-635c-4c3d-8c40-ea2db8aa2553 response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /omicron-config
10922025-04-30T06:42:39.749ZDEBGtest_snapshot_basic: client response result = Ok(Response { url: "http://[::1]:41923/omicron-config", status: 200, headers: {"content-type": "application/json", "x-request-id": "373aba86-635c-4c3d-8c40-ea2db8aa2553", "content-length": "26", "date": "Wed, 30 Apr 2025 06:42:39 GMT"} })
10932025-04-30T06:42:39.751ZDEBGtest_snapshot_basic: client request body = Some(Body) method = PUT uri = http://[::1]:41923/network-bootstore-config
10942025-04-30T06:42:39.760ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
10952025-04-30T06:42:39.770ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Fleet result = Ok(())
10962025-04-30T06:42:39.771ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database
10972025-04-30T06:42:39.771ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
10982025-04-30T06:42:39.773ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
10992025-04-30T06:42:39.773ZTRACtest_snapshot_basic (BackgroundTasks): webhook dispatching completed successfully background_task = webhook_dispatcher events_dispatched = 0 events_without_receivers = 0
11002025-04-30T06:42:39.773ZDEBGtest_snapshot_basic (BackgroundTasks): activation complete background_task = webhook_dispatcher elapsed = 1.257499637s iteration = 1 status = {"dispatched":[],"errors":[],"glob_version":"140.0.0","globs_reprocessed":{},"no_receivers":[]}
11012025-04-30T06:42:39.776ZTRACtest_snapshot_basic (dropshot): incoming request local_addr = [::1]:41923 method = PUT remote_addr = [::1]:34449 req_id = 834832ec-4c89-4eef-abd9-448a34b77620 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /network-bootstore-config
11022025-04-30T06:42:39.777ZINFOtest_snapshot_basic (dropshot): request completed latency_us = 793 local_addr = [::1]:41923 method = PUT remote_addr = [::1]:34449 req_id = 834832ec-4c89-4eef-abd9-448a34b77620 response_code = 204 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /network-bootstore-config
11032025-04-30T06:42:39.779ZDEBGtest_snapshot_basic: client response result = Ok(Response { url: "http://[::1]:41923/network-bootstore-config", status: 204, headers: {"x-request-id": "834832ec-4c89-4eef-abd9-448a34b77620", "date": "Wed, 30 Apr 2025 06:42:39 GMT"} })
11042025-04-30T06:42:39.780ZDEBGtest_snapshot_basic (init_with_steps): Running step start_nexus_external
11052025-04-30T06:42:39.780ZDEBGtest_snapshot_basic: Starting Nexus (external API)
11062025-04-30T06:42:42.660ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Fleet
11072025-04-30T06:42:42.660ZTRACtest_snapshot_basic (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
11082025-04-30T06:42:42.660ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
11092025-04-30T06:42:42.660ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {} }
11102025-04-30T06:42:42.662ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
11112025-04-30T06:42:42.676ZTRACtest_snapshot_basic (BackgroundTasks): no more instances in need of reincarnation background_task = instance_reincarnation reincarnation_reason = start saga failed total_found = 0
11122025-04-30T06:42:42.676ZINFOtest_snapshot_basic (BackgroundTasks): instance reincarnation completed successfully background_task = instance_reincarnation instances_changed_state = 0 instances_found = 0 instances_reincarnated = 0
11132025-04-30T06:42:42.677ZDEBGtest_snapshot_basic (BackgroundTasks): activation complete background_task = instance_reincarnation elapsed = 4.172207531s iteration = 1 status = {"changed_state":[],"disabled":false,"errors":[],"instances_found":{"Failed":0,"SagaUnwound":0},"instances_reincarnated":[],"restart_errors":[]}
11142025-04-30T06:42:42.677ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet
11152025-04-30T06:42:42.677ZTRACtest_snapshot_basic (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = v2p_manager resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
11162025-04-30T06:42:42.677ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database
11172025-04-30T06:42:42.677ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
11182025-04-30T06:42:42.678ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
11192025-04-30T06:42:42.762ZINFOtest_snapshot_basic (BackgroundTasks): VPC route manager sled b6d65341-167c-41df-9b5c-41cded99c229 background_task = vpc_route_manager
11202025-04-30T06:42:42.763ZDEBGtest_snapshot_basic (BackgroundTasks): client request SledAgent = b6d65341-167c-41df-9b5c-41cded99c229 background_task = vpc_route_manager body = None method = GET uri = http://[::1]:41923/vpc-routes
11212025-04-30T06:42:42.855ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = tuf_artifact_replication resource = Fleet
11222025-04-30T06:42:42.855ZTRACtest_snapshot_basic (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = tuf_artifact_replication resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
11232025-04-30T06:42:42.855ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = tuf_artifact_replication resource = Database
11242025-04-30T06:42:42.855ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = tuf_artifact_replication roles = RoleSet { roles: {} }
11252025-04-30T06:42:42.858ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = tuf_artifact_replication resource = Database result = Ok(())
11262025-04-30T06:42:42.861ZDEBGtest_snapshot_basic (BackgroundTasks): activation complete background_task = region_snapshot_replacement_step elapsed = 4.349594935s iteration = 1 status = {"errors":[],"step_garbage_collect_invoked_ok":[],"step_invoked_ok":[],"step_records_created_ok":[],"step_set_volume_deleted_ok":[]}
11272025-04-30T06:42:42.864ZTRACtest_snapshot_basic (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = service_firewall_rule_propagation resource_id = 001de000-5110-4000-8000-000000000001 resource_type = Silo
11282025-04-30T06:42:42.864ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database
11292025-04-30T06:42:42.864ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
11302025-04-30T06:42:42.865ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
11312025-04-30T06:42:42.865ZWARNtest_snapshot_basic (BackgroundTasks): failed to read Silo/DNS/TLS configuration background_task = external_endpoints error = Internal Error: expected at least one external DNS zone
11322025-04-30T06:42:42.866ZDEBGtest_snapshot_basic (BackgroundTasks): activation complete background_task = external_endpoints elapsed = 4.444801497s iteration = 1 status = {"error":"failed to read Silo/DNS/TLS configuration: Internal Error: expected at least one external DNS zone"}
11332025-04-30T06:42:42.866ZINFOtest_snapshot_basic (dropshot): accepted connection local_addr = [::1]:41923 remote_addr = [::1]:47066 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
11342025-04-30T06:42:42.870ZDEBGtest_snapshot_basic (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = []
11352025-04-30T06:42:42.870ZTRACtest_snapshot_basic (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
11362025-04-30T06:42:42.872ZTRACtest_snapshot_basic (dropshot): incoming request local_addr = [::1]:41923 method = GET remote_addr = [::1]:47066 req_id = decf5ae4-166f-434d-8d22-52284d7a67b5 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /vpc-routes
11372025-04-30T06:42:42.873ZINFOtest_snapshot_basic (dropshot): request completed latency_us = 918 local_addr = [::1]:41923 method = GET remote_addr = [::1]:47066 req_id = decf5ae4-166f-434d-8d22-52284d7a67b5 response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /vpc-routes
11382025-04-30T06:42:42.874ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
11392025-04-30T06:42:42.879ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Fleet result = Ok(())
11402025-04-30T06:42:42.879ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
11412025-04-30T06:42:42.879ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {} }
11422025-04-30T06:42:42.880ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
11432025-04-30T06:42:42.885ZTRACtest_snapshot_basic (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: 85 a1 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 ............._mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:58458 req_id = b2b598c6-95e3-4214-bb2e-608e20476433
11442025-04-30T06:42:42.885ZDEBGtest_snapshot_basic (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 34209,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:58458 req_id = b2b598c6-95e3-4214-bb2e-608e20476433
11452025-04-30T06:42:42.885ZDEBGtest_snapshot_basic (store): query key key = _mgs._tcp
11462025-04-30T06:42:42.886ZDEBGtest_snapshot_basic (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
11472025-04-30T06:42:42.886ZDEBGtest_snapshot_basic (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:58458 query = LowerQuery { name: LowerName(Name("_mgs._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 41502, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = b2b598c6-95e3-4214-bb2e-608e20476433
11482025-04-30T06:42:42.886ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
11492025-04-30T06:42:42.890ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
11502025-04-30T06:42:42.890ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet
11512025-04-30T06:42:42.890ZTRACtest_snapshot_basic (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = v2p_manager resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
11522025-04-30T06:42:42.890ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database
11532025-04-30T06:42:42.890ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
11542025-04-30T06:42:42.891ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
11552025-04-30T06:42:42.891ZDEBGtest_snapshot_basic (BackgroundTasks): client response SledAgent = b6d65341-167c-41df-9b5c-41cded99c229 background_task = vpc_route_manager result = Ok(Response { url: "http://[::1]:41923/vpc-routes", status: 200, headers: {"content-type": "application/json", "x-request-id": "decf5ae4-166f-434d-8d22-52284d7a67b5", "content-length": "2", "date": "Wed, 30 Apr 2025 06:42:42 GMT"} })
11562025-04-30T06:42:42.891ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database
11572025-04-30T06:42:42.891ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
11582025-04-30T06:42:42.896ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
11592025-04-30T06:42:42.952ZINFOtest_snapshot_basic (BackgroundTasks): No bundles to collect background_task = support_bundle_collector
11602025-04-30T06:42:42.952ZDEBGtest_snapshot_basic (BackgroundTasks): activation complete background_task = support_bundle_collector elapsed = 4.526440237s iteration = 1 status = {"cleanup_err":null,"cleanup_report":{"db_destroying_bundles_removed":0,"db_failing_bundles_updated":0,"sled_bundles_delete_failed":0,"sled_bundles_deleted_not_found":0,"sled_bundles_deleted_ok":0},"collection_err":null,"collection_report":null}
11612025-04-30T06:42:42.954ZDEBGtest_snapshot_basic (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 41502, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 4227, tv_nsec: 753312302 } })
11622025-04-30T06:42:42.954ZTRACtest_snapshot_basic (DnsResolver): lookup_service_targets: looking up SRV target
11632025-04-30T06:42:42.955ZTRACtest_snapshot_basic (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = service_firewall_rule_propagation resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
11642025-04-30T06:42:42.955ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database
11652025-04-30T06:42:42.955ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
11662025-04-30T06:42:42.956ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
11672025-04-30T06:42:42.956ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = tuf_artifact_replication roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
11682025-04-30T06:42:42.960ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = tuf_artifact_replication resource = Fleet result = Ok(())
11692025-04-30T06:42:42.960ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = tuf_artifact_replication resource = Database
11702025-04-30T06:42:42.960ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = tuf_artifact_replication roles = RoleSet { roles: {} }
11712025-04-30T06:42:42.961ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = tuf_artifact_replication resource = Database result = Ok(())
11722025-04-30T06:42:42.972ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
11732025-04-30T06:42:42.977ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
11742025-04-30T06:42:42.978ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database
11752025-04-30T06:42:42.978ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
11762025-04-30T06:42:42.978ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
11772025-04-30T06:42:42.979ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
11782025-04-30T06:42:42.979ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {} }
11792025-04-30T06:42:42.980ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
11802025-04-30T06:42:42.981ZTRACtest_snapshot_basic (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: 7c ed 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e |...........-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:59229 req_id = 380cd45a-2af6-4e13-aae1-e05815ff2c64
11812025-04-30T06:42:42.981ZDEBGtest_snapshot_basic (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 31981,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:59229 req_id = 380cd45a-2af6-4e13-aae1-e05815ff2c64
11822025-04-30T06:42:42.981ZDEBGtest_snapshot_basic (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
11832025-04-30T06:42:42.981ZDEBGtest_snapshot_basic (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:59229 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = 380cd45a-2af6-4e13-aae1-e05815ff2c64
11842025-04-30T06:42:42.983ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
11852025-04-30T06:42:42.993ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ById(001de000-074c-4000-8000-000000000000) } result = Ok(())
11862025-04-30T06:42:42.993ZTRACtest_snapshot_basic (BackgroundTasks): successful lookup of siloed resource "Vpc" using built-in user background_task = service_firewall_rule_propagation
11872025-04-30T06:42:42.993ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ById(001de000-074c-4000-8000-000000000000) }
11882025-04-30T06:42:42.993ZTRACtest_snapshot_basic (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = service_firewall_rule_propagation resource_id = 001de000-4401-4000-8000-000000000000 resource_type = Project
11892025-04-30T06:42:42.994ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database
11902025-04-30T06:42:42.994ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
11912025-04-30T06:42:42.997ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
11922025-04-30T06:42:43.083ZTRACtest_snapshot_basic (DnsResolver): lookup_all_socket_v6 srv dns_name = _clickhouse-admin-keeper._tcp.control-plane.oxide.internal
11932025-04-30T06:42:43.110ZINFOtest_snapshot_basic (BackgroundTasks): computed internet gateway mappings for sled assocs = {} background_task = vpc_route_manager sled = sim-b6d65341
11942025-04-30T06:42:43.110ZDEBGtest_snapshot_basic (BackgroundTasks): client request SledAgent = b6d65341-167c-41df-9b5c-41cded99c229 background_task = vpc_route_manager body = Some(Body) method = PUT uri = http://[::1]:41923/eip-gateways
11952025-04-30T06:42:43.110ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = tuf_artifact_replication resource = Fleet
11962025-04-30T06:42:43.111ZTRACtest_snapshot_basic (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = tuf_artifact_replication resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
11972025-04-30T06:42:43.111ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = tuf_artifact_replication resource = Database
11982025-04-30T06:42:43.111ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = tuf_artifact_replication roles = RoleSet { roles: {} }
11992025-04-30T06:42:43.113ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = tuf_artifact_replication resource = Database result = Ok(())
12002025-04-30T06:42:43.115ZTRACtest_snapshot_basic (dropshot): incoming request local_addr = [::1]:41923 method = PUT remote_addr = [::1]:47066 req_id = 7631d22d-856e-4916-be82-aeae6db379ce sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /eip-gateways
12012025-04-30T06:42:43.119ZTRACtest_snapshot_basic (dns): buffer buffer = Length: 87 (0x57) bytes\n0000: f6 ea 01 00 00 01 00 00 00 00 00 01 18 5f 63 6c ............._cl\n0010: 69 63 6b 68 6f 75 73 65 2d 61 64 6d 69 6e 2d 6b ickhouse-admin-k\n0020: 65 65 70 65 72 04 5f 74 63 70 0d 63 6f 6e 74 72 eeper._tcp.contr\n0030: 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 69 ol-plane.oxide.i\n0040: 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 04 nternal..!....).\n0050: d0 00 00 00 00 00 00 ....... kind = dns peer_addr = [::1]:60084 req_id = a05566e2-65e3-4dd0-9289-9cc82b6e62a6
12022025-04-30T06:42:43.119ZDEBGtest_snapshot_basic (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 63210,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 24,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 97,\n 100,\n 109,\n 105,\n 110,\n 45,\n 107,\n 101,\n 101,\n 112,\n 101,\n 114,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:60084 req_id = a05566e2-65e3-4dd0-9289-9cc82b6e62a6
12032025-04-30T06:42:43.120ZDEBGtest_snapshot_basic (store): query key key = _clickhouse-admin-keeper._tcp
12042025-04-30T06:42:43.120ZERROtest_snapshot_basic (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 63210, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } NXDOMAIN: no records found for name: "_clickhouse-admin-keeper._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:60084 req_id = a05566e2-65e3-4dd0-9289-9cc82b6e62a6
12052025-04-30T06:42:43.121ZINFOtest_snapshot_basic (dropshot): request completed latency_us = 5461 local_addr = [::1]:41923 method = PUT remote_addr = [::1]:47066 req_id = 7631d22d-856e-4916-be82-aeae6db379ce response_code = 204 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /eip-gateways
12062025-04-30T06:42:43.121ZTRACtest_snapshot_basic (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = service_firewall_rule_propagation resource_id = 001de000-5110-4000-8000-000000000001 resource_type = Silo
12072025-04-30T06:42:43.121ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database
12082025-04-30T06:42:43.121ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
12092025-04-30T06:42:43.122ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
12102025-04-30T06:42:43.122ZINFOtest_snapshot_basic (InternalApi): set cockroachdb setting setting = cluster.preserve_downgrade_option value = 22.1
12112025-04-30T06:42:43.122ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
12122025-04-30T06:42:43.122ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {} }
12132025-04-30T06:42:43.123ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
12142025-04-30T06:42:43.124ZDEBGtest_snapshot_basic (BackgroundTasks): begin collection background_task = inventory_collection
12152025-04-30T06:42:43.124ZDEBGtest_snapshot_basic (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:41502
12162025-04-30T06:42:43.124ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/ignition
12172025-04-30T06:42:43.124ZDEBGtest_snapshot_basic (BackgroundTasks): client response SledAgent = b6d65341-167c-41df-9b5c-41cded99c229 background_task = vpc_route_manager result = Ok(Response { url: "http://[::1]:41923/eip-gateways", status: 204, headers: {"x-request-id": "7631d22d-856e-4916-be82-aeae6db379ce", "date": "Wed, 30 Apr 2025 06:42:42 GMT"} })
12182025-04-30T06:42:43.125ZDEBGtest_snapshot_basic (BackgroundTasks): activation complete background_task = vpc_route_manager elapsed = 4.618516593s iteration = 1 status = {}
12192025-04-30T06:42:43.126ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = tuf_artifact_replication roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
12202025-04-30T06:42:43.132ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = tuf_artifact_replication resource = Fleet result = Ok(())
12212025-04-30T06:42:43.132ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = tuf_artifact_replication resource = Database
12222025-04-30T06:42:43.132ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = tuf_artifact_replication roles = RoleSet { roles: {} }
12232025-04-30T06:42:43.133ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = tuf_artifact_replication resource = Database result = Ok(())
12242025-04-30T06:42:43.133ZTRACtest_snapshot_basic (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = service_firewall_rule_propagation resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
12252025-04-30T06:42:43.133ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database
12262025-04-30T06:42:43.133ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
12272025-04-30T06:42:43.134ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
12282025-04-30T06:42:43.137ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
12292025-04-30T06:42:43.148ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ById(001de000-074c-4000-8000-000000000000) } result = Ok(())
12302025-04-30T06:42:43.148ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database
12312025-04-30T06:42:43.148ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
12322025-04-30T06:42:43.149ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
12332025-04-30T06:42:43.151ZDEBGtest_snapshot_basic (BackgroundTasks): client request SledAgent = b6d65341-167c-41df-9b5c-41cded99c229 background_task = tuf_artifact_replication body = Some(Body) method = PUT uri = http://[::1]:41923/artifacts-config
12342025-04-30T06:42:43.152ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Rack { parent: Fleet, key: c19a698f-c6f9-4a17-ae30-20d711b8f7dc, lookup_type: ById(c19a698f-c6f9-4a17-ae30-20d711b8f7dc) }
12352025-04-30T06:42:43.152ZTRACtest_snapshot_basic (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
12362025-04-30T06:42:43.152ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
12372025-04-30T06:42:43.152ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {} }
12382025-04-30T06:42:43.153ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
12392025-04-30T06:42:43.154ZINFOtest_snapshot_basic (dropshot): accepted connection local_addr = [::1]:41923 remote_addr = [::1]:54022 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
12402025-04-30T06:42:43.155ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database
12412025-04-30T06:42:43.155ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
12422025-04-30T06:42:43.156ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
12432025-04-30T06:42:43.158ZTRACtest_snapshot_basic (dropshot): incoming request local_addr = [::1]:41923 method = PUT remote_addr = [::1]:54022 req_id = 37baf881-e240-44d2-9fda-7c1f9812e698 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /artifacts-config
12442025-04-30T06:42:43.162ZINFOtest_snapshot_basic (ArtifactStore): No ledger in /var/tmp/omicron_tmp/.tmp7W42qF/artifact-config.json server = b6d65341-167c-41df-9b5c-41cded99c229 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
12452025-04-30T06:42:43.162ZDEBGtest_snapshot_basic (ArtifactStore): Failed to read ledger: Not found in storage path = /var/tmp/omicron_tmp/.tmp7W42qF/artifact-config.json server = b6d65341-167c-41df-9b5c-41cded99c229 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
12462025-04-30T06:42:43.162ZINFOtest_snapshot_basic (ArtifactStore): No ledger in /var/tmp/omicron_tmp/.tmpaLyh3W/artifact-config.json server = b6d65341-167c-41df-9b5c-41cded99c229 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
12472025-04-30T06:42:43.162ZDEBGtest_snapshot_basic (ArtifactStore): Failed to read ledger: Not found in storage path = /var/tmp/omicron_tmp/.tmpaLyh3W/artifact-config.json server = b6d65341-167c-41df-9b5c-41cded99c229 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
12482025-04-30T06:42:43.162ZINFOtest_snapshot_basic (ArtifactStore): Writing ledger to /var/tmp/omicron_tmp/.tmp7W42qF/.artifact-config.json.tmp server = b6d65341-167c-41df-9b5c-41cded99c229 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
12492025-04-30T06:42:43.162ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
12502025-04-30T06:42:43.171ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Rack { parent: Fleet, key: c19a698f-c6f9-4a17-ae30-20d711b8f7dc, lookup_type: ById(c19a698f-c6f9-4a17-ae30-20d711b8f7dc) } result = Ok(())
12512025-04-30T06:42:43.171ZDEBGtest_snapshot_basic (InternalApi): updating rack subnet for rack c19a698f-c6f9-4a17-ae30-20d711b8f7dc to Some( V6( Ipv6Network { addr: fd00:1122:3344:100::, prefix: 56, }, ), )
12522025-04-30T06:42:43.171ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
12532025-04-30T06:42:43.171ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {} }
12542025-04-30T06:42:43.172ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
12552025-04-30T06:42:43.172ZINFOtest_snapshot_basic (ArtifactStore): Writing ledger to /var/tmp/omicron_tmp/.tmpaLyh3W/.artifact-config.json.tmp server = b6d65341-167c-41df-9b5c-41cded99c229 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
12562025-04-30T06:42:43.180ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database
12572025-04-30T06:42:43.180ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
12582025-04-30T06:42:43.181ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
12592025-04-30T06:42:43.182ZINFOtest_snapshot_basic (ArtifactStore): Starting delete reconciler generation = 1 server = b6d65341-167c-41df-9b5c-41cded99c229 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
12602025-04-30T06:42:43.182ZINFOtest_snapshot_basic (dropshot): request completed latency_us = 24446 local_addr = [::1]:41923 method = PUT remote_addr = [::1]:54022 req_id = 37baf881-e240-44d2-9fda-7c1f9812e698 response_code = 204 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /artifacts-config
12612025-04-30T06:42:43.261ZDEBGtest_snapshot_basic (BackgroundTasks): client request SledAgent = b6d65341-167c-41df-9b5c-41cded99c229 background_task = v2p_manager body = None method = GET uri = http://[::1]:41923/v2p
12622025-04-30T06:42:43.262ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database
12632025-04-30T06:42:43.262ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
12642025-04-30T06:42:43.263ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
12652025-04-30T06:42:43.266ZINFOtest_snapshot_basic (dropshot): accepted connection local_addr = [::1]:41923 remote_addr = [::1]:54892 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
12662025-04-30T06:42:43.266ZDEBGtest_snapshot_basic (BackgroundTasks): client response SledAgent = b6d65341-167c-41df-9b5c-41cded99c229 background_task = tuf_artifact_replication result = Ok(Response { url: "http://[::1]:41923/artifacts-config", status: 204, headers: {"x-request-id": "37baf881-e240-44d2-9fda-7c1f9812e698", "date": "Wed, 30 Apr 2025 06:42:42 GMT"} })
12672025-04-30T06:42:43.266ZDEBGtest_snapshot_basic (BackgroundTasks): client request SledAgent = b6d65341-167c-41df-9b5c-41cded99c229 background_task = tuf_artifact_replication body = None method = GET uri = http://[::1]:41923/artifacts
12682025-04-30T06:42:43.267ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "41cf2cd6-1aa7-4d41-a509-d2e57cfbc389", "content-length": "835", "date": "Wed, 30 Apr 2025 06:42:42 GMT"} })
12692025-04-30T06:42:43.269ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/0
12702025-04-30T06:42:43.270ZINFOtest_snapshot_basic (InternalApi): Using static configuration for external switchports
12712025-04-30T06:42:43.270ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
12722025-04-30T06:42:43.270ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {} }
12732025-04-30T06:42:43.274ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
12742025-04-30T06:42:43.274ZTRACtest_snapshot_basic (dropshot): incoming request local_addr = [::1]:41923 method = GET remote_addr = [::1]:54022 req_id = 8aa4af77-ffed-49a2-9534-6da530a849db sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /artifacts
12752025-04-30T06:42:43.275ZTRACtest_snapshot_basic (dropshot): incoming request local_addr = [::1]:41923 method = GET remote_addr = [::1]:54892 req_id = 06081b32-2c1c-483f-9fb4-d874149cbfe0 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /v2p
12762025-04-30T06:42:43.287ZINFOtest_snapshot_basic (dropshot): request completed latency_us = 12672 local_addr = [::1]:41923 method = GET remote_addr = [::1]:54022 req_id = 8aa4af77-ffed-49a2-9534-6da530a849db response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /artifacts
12772025-04-30T06:42:43.287ZINFOtest_snapshot_basic (dropshot): request completed latency_us = 12456 local_addr = [::1]:41923 method = GET remote_addr = [::1]:54892 req_id = 06081b32-2c1c-483f-9fb4-d874149cbfe0 response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /v2p
12782025-04-30T06:42:43.287ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ById(001de000-074c-4000-8000-000000000000) }
12792025-04-30T06:42:43.287ZTRACtest_snapshot_basic (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = service_firewall_rule_propagation resource_id = 001de000-4401-4000-8000-000000000000 resource_type = Project
12802025-04-30T06:42:43.287ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database
12812025-04-30T06:42:43.287ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
12822025-04-30T06:42:43.290ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
12832025-04-30T06:42:43.292ZDEBGtest_snapshot_basic (BackgroundTasks): client response SledAgent = b6d65341-167c-41df-9b5c-41cded99c229 background_task = tuf_artifact_replication result = Ok(Response { url: "http://[::1]:41923/artifacts", status: 200, headers: {"content-type": "application/json", "x-request-id": "8aa4af77-ffed-49a2-9534-6da530a849db", "content-length": "26", "date": "Wed, 30 Apr 2025 06:42:42 GMT"} })
12842025-04-30T06:42:43.293ZINFOtest_snapshot_basic (BackgroundTasks): Successfully got artifact list background_task = tuf_artifact_replication sled = http://[::1]:41923
12852025-04-30T06:42:43.293ZDEBGtest_snapshot_basic (BackgroundTasks): client response SledAgent = b6d65341-167c-41df-9b5c-41cded99c229 background_task = v2p_manager result = Ok(Response { url: "http://[::1]:41923/v2p", status: 200, headers: {"content-type": "application/json", "x-request-id": "06081b32-2c1c-483f-9fb4-d874149cbfe0", "content-length": "2", "date": "Wed, 30 Apr 2025 06:42:42 GMT"} })
12862025-04-30T06:42:43.294ZINFOtest_snapshot_basic (BackgroundTasks): found opte v2p mappings background_task = v2p_manager interfaces = {} sled = sim-b6d65341
12872025-04-30T06:42:43.294ZINFOtest_snapshot_basic (BackgroundTasks): v2p mappings to delete background_task = v2p_manager mappings = [] sled = sim-b6d65341
12882025-04-30T06:42:43.294ZINFOtest_snapshot_basic (BackgroundTasks): v2p mappings to add background_task = v2p_manager mappings = [] sled = sim-b6d65341
12892025-04-30T06:42:43.296ZDEBGtest_snapshot_basic (BackgroundTasks): activation complete background_task = v2p_manager elapsed = 4.794744246s iteration = 1 status = {}
12902025-04-30T06:42:43.299ZDEBGtest_snapshot_basic (BackgroundTasks): activation complete background_task = tuf_artifact_replication elapsed = 4.784713951s iteration = 1 status = {"generation":1,"last_run_counters":{"copy_err":0,"copy_ok":0,"list_err":0,"list_ok":1,"put_config_err":0,"put_config_ok":1,"put_err":0,"put_ok":0},"lifetime_counters":{"copy_err":0,"copy_ok":0,"list_err":0,"list_ok":1,"put_config_err":0,"put_config_ok":1,"put_err":0,"put_ok":0},"local_repos":0,"request_debug_ringbuf":[{"generation":1,"operation":"put_config","target_sled":"b6d65341-167c-41df-9b5c-41cded99c229","time":"2025-04-30T06:42:43.266837673Z"},{"operation":"list","target_sled":"b6d65341-167c-41df-9b5c-41cded99c229","time":"2025-04-30T06:42:43.293070531Z"}]}
12912025-04-30T06:42:43.307ZTRACtest_snapshot_basic (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = service_firewall_rule_propagation resource_id = 001de000-5110-4000-8000-000000000001 resource_type = Silo
12922025-04-30T06:42:43.307ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database
12932025-04-30T06:42:43.307ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
12942025-04-30T06:42:43.308ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
12952025-04-30T06:42:43.315ZTRACtest_snapshot_basic (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = service_firewall_rule_propagation resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
12962025-04-30T06:42:43.315ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database
12972025-04-30T06:42:43.315ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
12982025-04-30T06:42:43.316ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
12992025-04-30T06:42:43.333ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
13002025-04-30T06:42:43.347ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ById(001de000-074c-4000-8000-000000000000) } result = Ok(())
13012025-04-30T06:42:43.347ZTRACtest_snapshot_basic (BackgroundTasks): successful lookup of siloed resource "Vpc" using built-in user background_task = service_firewall_rule_propagation
13022025-04-30T06:42:43.347ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database
13032025-04-30T06:42:43.347ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
13042025-04-30T06:42:43.348ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
13052025-04-30T06:42:43.360ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "674179e1-baf9-4c7e-9e39-d4a9461f20e6", "content-length": "734", "date": "Wed, 30 Apr 2025 06:42:42 GMT"} })
13062025-04-30T06:42:43.361ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/0/component/sp/caboose?firmware_slot=0
13072025-04-30T06:42:43.363ZINFOtest_snapshot_basic (InternalApi): Recording Rack Network Configuration
13082025-04-30T06:42:43.363ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
13092025-04-30T06:42:43.363ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {} }
13102025-04-30T06:42:43.364ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
13112025-04-30T06:42:43.370ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database
13122025-04-30T06:42:43.371ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
13132025-04-30T06:42:43.373ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
13142025-04-30T06:42:43.380ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database
13152025-04-30T06:42:43.380ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
13162025-04-30T06:42:43.381ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
13172025-04-30T06:42:43.385ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database
13182025-04-30T06:42:43.385ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
13192025-04-30T06:42:43.386ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
13202025-04-30T06:42:43.390ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = VpcSubnet { parent: Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ByName("oxide-services") }, key: 001de000-c470-4000-8000-000000000001, lookup_type: ByName("external-dns") }
13212025-04-30T06:42:43.390ZTRACtest_snapshot_basic (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = service_firewall_rule_propagation resource_id = 001de000-4401-4000-8000-000000000000 resource_type = Project
13222025-04-30T06:42:43.390ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database
13232025-04-30T06:42:43.390ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
13242025-04-30T06:42:43.393ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
13252025-04-30T06:42:43.407ZTRACtest_snapshot_basic (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = service_firewall_rule_propagation resource_id = 001de000-5110-4000-8000-000000000001 resource_type = Silo
13262025-04-30T06:42:43.407ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database
13272025-04-30T06:42:43.407ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
13282025-04-30T06:42:43.407ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
13292025-04-30T06:42:43.414ZTRACtest_snapshot_basic (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = service_firewall_rule_propagation resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
13302025-04-30T06:42:43.414ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database
13312025-04-30T06:42:43.414ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
13322025-04-30T06:42:43.414ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
13332025-04-30T06:42:43.419ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
13342025-04-30T06:42:43.431ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = VpcSubnet { parent: Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ByName("oxide-services") }, key: 001de000-c470-4000-8000-000000000001, lookup_type: ByName("external-dns") } result = Ok(())
13352025-04-30T06:42:43.431ZTRACtest_snapshot_basic (BackgroundTasks): successful lookup of siloed resource "VpcSubnet" using built-in user background_task = service_firewall_rule_propagation
13362025-04-30T06:42:43.431ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = VpcSubnet { parent: Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ByName("oxide-services") }, key: 001de000-c470-4000-8000-000000000001, lookup_type: ByName("external-dns") }
13372025-04-30T06:42:43.431ZTRACtest_snapshot_basic (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = service_firewall_rule_propagation resource_id = 001de000-4401-4000-8000-000000000000 resource_type = Project
13382025-04-30T06:42:43.431ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database
13392025-04-30T06:42:43.432ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
13402025-04-30T06:42:43.433ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
13412025-04-30T06:42:43.438ZTRACtest_snapshot_basic (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = service_firewall_rule_propagation resource_id = 001de000-5110-4000-8000-000000000001 resource_type = Silo
13422025-04-30T06:42:43.438ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database
13432025-04-30T06:42:43.438ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
13442025-04-30T06:42:43.440ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
13452025-04-30T06:42:43.442ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "032df1d6-5dc9-4f7a-8bdd-6951069dd7ff", "content-length": "111", "date": "Wed, 30 Apr 2025 06:42:42 GMT"} })
13462025-04-30T06:42:43.444ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/0/component/sp/caboose?firmware_slot=1
13472025-04-30T06:42:43.452ZTRACtest_snapshot_basic (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = service_firewall_rule_propagation resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
13482025-04-30T06:42:43.452ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database
13492025-04-30T06:42:43.452ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
13502025-04-30T06:42:43.453ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
13512025-04-30T06:42:43.468ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
13522025-04-30T06:42:43.479ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = VpcSubnet { parent: Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ByName("oxide-services") }, key: 001de000-c470-4000-8000-000000000001, lookup_type: ByName("external-dns") } result = Ok(())
13532025-04-30T06:42:43.480ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database
13542025-04-30T06:42:43.480ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
13552025-04-30T06:42:43.481ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
13562025-04-30T06:42:43.493ZTRACtest_snapshot_basic (InternalApi): authorize begin action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Fleet
13572025-04-30T06:42:43.493ZTRACtest_snapshot_basic (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
13582025-04-30T06:42:43.493ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
13592025-04-30T06:42:43.494ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {} }
13602025-04-30T06:42:43.495ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
13612025-04-30T06:42:43.498ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database
13622025-04-30T06:42:43.499ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
13632025-04-30T06:42:43.499ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
13642025-04-30T06:42:43.506ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
13652025-04-30T06:42:43.510ZDEBGtest_snapshot_basic (InternalApi): authorize result action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Fleet result = Ok(())
13662025-04-30T06:42:43.510ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
13672025-04-30T06:42:43.510ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {} }
13682025-04-30T06:42:43.510ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
13692025-04-30T06:42:43.512ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database
13702025-04-30T06:42:43.512ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
13712025-04-30T06:42:43.512ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
13722025-04-30T06:42:43.517ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database
13732025-04-30T06:42:43.517ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
13742025-04-30T06:42:43.526ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
13752025-04-30T06:42:43.526ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = IpPool { parent: Fleet, key: 81c37ead-f093-42c9-b478-64803a43b594, lookup_type: ByName("oxide-service-pool") }
13762025-04-30T06:42:43.526ZTRACtest_snapshot_basic (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
13772025-04-30T06:42:43.526ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
13782025-04-30T06:42:43.526ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {} }
13792025-04-30T06:42:43.529ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
13802025-04-30T06:42:43.532ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database
13812025-04-30T06:42:43.532ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
13822025-04-30T06:42:43.533ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
13832025-04-30T06:42:43.535ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
13842025-04-30T06:42:43.540ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = IpPool { parent: Fleet, key: 81c37ead-f093-42c9-b478-64803a43b594, lookup_type: ByName("oxide-service-pool") } result = Ok(())
13852025-04-30T06:42:43.540ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
13862025-04-30T06:42:43.540ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {} }
13872025-04-30T06:42:43.541ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
13882025-04-30T06:42:43.545ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = VpcSubnet { parent: Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ByName("oxide-services") }, key: 001de000-c470-4000-8000-000000000002, lookup_type: ByName("nexus") }
13892025-04-30T06:42:43.545ZTRACtest_snapshot_basic (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = service_firewall_rule_propagation resource_id = 001de000-4401-4000-8000-000000000000 resource_type = Project
13902025-04-30T06:42:43.545ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database
13912025-04-30T06:42:43.545ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
13922025-04-30T06:42:43.546ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
13932025-04-30T06:42:43.555ZTRACtest_snapshot_basic (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = service_firewall_rule_propagation resource_id = 001de000-5110-4000-8000-000000000001 resource_type = Silo
13942025-04-30T06:42:43.555ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database
13952025-04-30T06:42:43.555ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
13962025-04-30T06:42:43.556ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
13972025-04-30T06:42:43.556ZTRACtest_snapshot_basic (InternalApi): authorize begin action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = IpPool { parent: Fleet, key: 81c37ead-f093-42c9-b478-64803a43b594, lookup_type: ByName("oxide-service-pool") }
13982025-04-30T06:42:43.556ZTRACtest_snapshot_basic (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
13992025-04-30T06:42:43.557ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
14002025-04-30T06:42:43.557ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {} }
14012025-04-30T06:42:43.557ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
14022025-04-30T06:42:43.562ZTRACtest_snapshot_basic (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = service_firewall_rule_propagation resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
14032025-04-30T06:42:43.562ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database
14042025-04-30T06:42:43.562ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
14052025-04-30T06:42:43.563ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
14062025-04-30T06:42:43.565ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
14072025-04-30T06:42:43.571ZDEBGtest_snapshot_basic (InternalApi): authorize result action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = IpPool { parent: Fleet, key: 81c37ead-f093-42c9-b478-64803a43b594, lookup_type: ByName("oxide-service-pool") } result = Ok(())
14082025-04-30T06:42:43.577ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
14092025-04-30T06:42:43.602ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = VpcSubnet { parent: Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ByName("oxide-services") }, key: 001de000-c470-4000-8000-000000000002, lookup_type: ByName("nexus") } result = Ok(())
14102025-04-30T06:42:43.603ZTRACtest_snapshot_basic (BackgroundTasks): successful lookup of siloed resource "VpcSubnet" using built-in user background_task = service_firewall_rule_propagation
14112025-04-30T06:42:43.603ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = VpcSubnet { parent: Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ByName("oxide-services") }, key: 001de000-c470-4000-8000-000000000002, lookup_type: ByName("nexus") }
14122025-04-30T06:42:43.603ZTRACtest_snapshot_basic (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = service_firewall_rule_propagation resource_id = 001de000-4401-4000-8000-000000000000 resource_type = Project
14132025-04-30T06:42:43.603ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database
14142025-04-30T06:42:43.603ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
14152025-04-30T06:42:43.604ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
14162025-04-30T06:42:43.605ZTRACtest_snapshot_basic (InternalApi): authorize begin action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = IpPool { parent: Fleet, key: 81c37ead-f093-42c9-b478-64803a43b594, lookup_type: ByName("oxide-service-pool") }
14172025-04-30T06:42:43.605ZTRACtest_snapshot_basic (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
14182025-04-30T06:42:43.605ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
14192025-04-30T06:42:43.605ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {} }
14202025-04-30T06:42:43.606ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
14212025-04-30T06:42:43.613ZTRACtest_snapshot_basic (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = service_firewall_rule_propagation resource_id = 001de000-5110-4000-8000-000000000001 resource_type = Silo
14222025-04-30T06:42:43.613ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database
14232025-04-30T06:42:43.613ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
14242025-04-30T06:42:43.617ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
14252025-04-30T06:42:43.617ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
14262025-04-30T06:42:43.624ZDEBGtest_snapshot_basic (InternalApi): authorize result action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = IpPool { parent: Fleet, key: 81c37ead-f093-42c9-b478-64803a43b594, lookup_type: ByName("oxide-service-pool") } result = Ok(())
14272025-04-30T06:42:43.626ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "f1a3e152-2532-434b-9a4f-3896968d7578", "content-length": "111", "date": "Wed, 30 Apr 2025 06:42:42 GMT"} })
14282025-04-30T06:42:43.626ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/0/component/rot/caboose?firmware_slot=0
14292025-04-30T06:42:43.636ZTRACtest_snapshot_basic (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = service_firewall_rule_propagation resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
14302025-04-30T06:42:43.636ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database
14312025-04-30T06:42:43.636ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
14322025-04-30T06:42:43.637ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
14332025-04-30T06:42:43.638ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = BlueprintConfig
14342025-04-30T06:42:43.638ZTRACtest_snapshot_basic (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
14352025-04-30T06:42:43.638ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
14362025-04-30T06:42:43.638ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {} }
14372025-04-30T06:42:43.639ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
14382025-04-30T06:42:43.645ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
14392025-04-30T06:42:43.654ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = VpcSubnet { parent: Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ByName("oxide-services") }, key: 001de000-c470-4000-8000-000000000002, lookup_type: ByName("nexus") } result = Ok(())
14402025-04-30T06:42:43.654ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database
14412025-04-30T06:42:43.654ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
14422025-04-30T06:42:43.654ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
14432025-04-30T06:42:43.656ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
14442025-04-30T06:42:43.664ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = BlueprintConfig result = Ok(())
14452025-04-30T06:42:43.674ZDEBGtest_snapshot_basic (service-firewall-rule-progator): resolved names for firewall rules background_task = service_firewall_rule_propagation instance_interfaces = {} subnet_interfaces = {} subnet_networks = {Name("external-dns"): [V4(Ipv4Network { addr: 172.30.1.0, prefix: 24 }), V6(Ipv6Network { addr: fd77:e9d2:9cd9:1::, prefix: 64 })], Name("nexus"): [V4(Ipv4Network { addr: 172.30.2.0, prefix: 24 }), V6(Ipv6Network { addr: fd77:e9d2:9cd9:2::, prefix: 64 })]} vpc_interfaces = {}
14462025-04-30T06:42:43.674ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Fleet
14472025-04-30T06:42:43.674ZTRACtest_snapshot_basic (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = service_firewall_rule_propagation resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
14482025-04-30T06:42:43.674ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database
14492025-04-30T06:42:43.674ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
14502025-04-30T06:42:43.675ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
14512025-04-30T06:42:43.681ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
14522025-04-30T06:42:43.685ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Fleet result = Ok(())
14532025-04-30T06:42:43.685ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database
14542025-04-30T06:42:43.685ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
14552025-04-30T06:42:43.685ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
14562025-04-30T06:42:43.687ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "c85e8817-251a-4898-9024-fc15082d6c27", "content-length": "170", "date": "Wed, 30 Apr 2025 06:42:42 GMT"} })
14572025-04-30T06:42:43.688ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/0/component/rot/caboose?firmware_slot=1
14582025-04-30T06:42:43.701ZTRACtest_snapshot_basic (service-firewall-rule-progator): fetched allowlist from DB allowed = AllowList { id: 001de000-a110-4000-8000-000000000000, time_created: 2025-04-30T06:06:08.248500Z, time_modified: 2025-04-30T06:06:08.248500Z, allowed_ips: None } background_task = service_firewall_rule_propagation
14592025-04-30T06:42:43.701ZDEBGtest_snapshot_basic (service-firewall-rule-progator): Allowlist for user-facing services is set to allow any inbound traffic. Existing VPC firewall rules will not be modified. background_task = service_firewall_rule_propagation
14602025-04-30T06:42:43.701ZDEBGtest_snapshot_basic (service-firewall-rule-progator): resolved firewall rules for sled agents background_task = service_firewall_rule_propagation sled_agent_rules = []
14612025-04-30T06:42:43.701ZDEBGtest_snapshot_basic (service-firewall-rule-progator): resolved 0 rules for sleds background_task = service_firewall_rule_propagation
14622025-04-30T06:42:43.765ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "3839145e-ac08-41d6-8d02-a30359072a08", "content-length": "170", "date": "Wed, 30 Apr 2025 06:42:42 GMT"} })
14632025-04-30T06:42:43.766ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/0/component/stage0/caboose?firmware_slot=0
14642025-04-30T06:42:43.768ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "78bfda69-f9fb-478b-92bf-1e3067668791", "content-length": "179", "date": "Wed, 30 Apr 2025 06:42:42 GMT"} })
14652025-04-30T06:42:43.768ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/0/component/stage0/caboose?firmware_slot=1
14662025-04-30T06:42:43.773ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "e364bc82-4a0e-4faa-baa5-621c9cfe9069", "content-length": "179", "date": "Wed, 30 Apr 2025 06:42:42 GMT"} })
14672025-04-30T06:42:43.773ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/0/component/rot/cmpa
14682025-04-30T06:42:43.775ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "46c41c2a-c25c-4141-b5d3-4ba25739ceee", "content-length": "702", "date": "Wed, 30 Apr 2025 06:42:42 GMT"} })
14692025-04-30T06:42:43.776ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/0/component/rot/cfpa
14702025-04-30T06:42:43.785ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "98545297-6bab-4260-b32b-5d4d4372498a", "content-length": "727", "date": "Wed, 30 Apr 2025 06:42:42 GMT"} })
14712025-04-30T06:42:43.785ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/0/component/rot/cfpa
14722025-04-30T06:42:43.795ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "5dbc1e9e-7e31-45b4-8a78-6b9db826cc98", "content-length": "729", "date": "Wed, 30 Apr 2025 06:42:42 GMT"} })
14732025-04-30T06:42:43.795ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/0/component/rot/cfpa
14742025-04-30T06:42:43.796ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "d321e164-0faf-44de-be06-ec87c77e45b0", "content-length": "728", "date": "Wed, 30 Apr 2025 06:42:42 GMT"} })
14752025-04-30T06:42:43.797ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/1
14762025-04-30T06:42:43.798ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "6d0e3597-fb3e-4c78-a8dd-92e896eb2131", "content-length": "734", "date": "Wed, 30 Apr 2025 06:42:42 GMT"} })
14772025-04-30T06:42:43.798ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/1/component/sp/caboose?firmware_slot=0
14782025-04-30T06:42:43.801ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "d522fa8f-50df-4463-b8fb-5df98962f72b", "content-length": "111", "date": "Wed, 30 Apr 2025 06:42:42 GMT"} })
14792025-04-30T06:42:43.801ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/1/component/sp/caboose?firmware_slot=1
14802025-04-30T06:42:43.804ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "3b4dabfd-1bcc-4079-9217-9dd750a23279", "content-length": "111", "date": "Wed, 30 Apr 2025 06:42:42 GMT"} })
14812025-04-30T06:42:43.804ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/1/component/rot/caboose?firmware_slot=0
14822025-04-30T06:42:43.807ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "4ff80b61-f8f6-42ca-8865-69ec7f1c0ab7", "content-length": "170", "date": "Wed, 30 Apr 2025 06:42:42 GMT"} })
14832025-04-30T06:42:43.808ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/1/component/rot/caboose?firmware_slot=1
14842025-04-30T06:42:43.810ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "a5d29b3b-da82-48a6-9f93-5fe2ac88ea9c", "content-length": "170", "date": "Wed, 30 Apr 2025 06:42:42 GMT"} })
14852025-04-30T06:42:43.810ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/1/component/stage0/caboose?firmware_slot=0
14862025-04-30T06:42:43.812ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "0ad7004e-5984-468a-b871-cfeacb2c122c", "content-length": "179", "date": "Wed, 30 Apr 2025 06:42:42 GMT"} })
14872025-04-30T06:42:43.812ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/1/component/stage0/caboose?firmware_slot=1
14882025-04-30T06:42:43.814ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "8fc381df-825e-4797-92dd-d12b7b2c586f", "content-length": "179", "date": "Wed, 30 Apr 2025 06:42:42 GMT"} })
14892025-04-30T06:42:43.814ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/1/component/rot/cmpa
14902025-04-30T06:42:43.815ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "4914713b-9924-4dc5-8eb7-21142b522772", "content-length": "702", "date": "Wed, 30 Apr 2025 06:42:42 GMT"} })
14912025-04-30T06:42:43.815ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/1/component/rot/cfpa
14922025-04-30T06:42:43.816ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "045811d3-73b6-4bc4-b4d9-ce07fdee2081", "content-length": "727", "date": "Wed, 30 Apr 2025 06:42:42 GMT"} })
14932025-04-30T06:42:43.816ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/1/component/rot/cfpa
14942025-04-30T06:42:43.817ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "358aecae-057e-4b06-93ad-4fe672e4cf23", "content-length": "729", "date": "Wed, 30 Apr 2025 06:42:42 GMT"} })
14952025-04-30T06:42:43.817ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/1/component/rot/cfpa
14962025-04-30T06:42:43.821ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "54d55a3a-3b6b-4450-8f44-822773f7bc32", "content-length": "728", "date": "Wed, 30 Apr 2025 06:42:42 GMT"} })
14972025-04-30T06:42:43.821ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/0
14982025-04-30T06:42:43.824ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "cf805ab5-bced-48ff-a65b-13f1465722d8", "content-length": "723", "date": "Wed, 30 Apr 2025 06:42:42 GMT"} })
14992025-04-30T06:42:43.825ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/0/component/sp/caboose?firmware_slot=0
15002025-04-30T06:42:43.827ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "b0feb3ed-9a3e-4302-b78d-992c49cfe283", "content-length": "109", "date": "Wed, 30 Apr 2025 06:42:42 GMT"} })
15012025-04-30T06:42:43.828ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/0/component/sp/caboose?firmware_slot=1
15022025-04-30T06:42:43.830ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "024e8646-8a9b-41a1-9c8a-f41e62d5a7bd", "content-length": "109", "date": "Wed, 30 Apr 2025 06:42:42 GMT"} })
15032025-04-30T06:42:43.830ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/0/component/rot/caboose?firmware_slot=0
15042025-04-30T06:42:43.831ZINFOtest_snapshot_basic (InternalApi): inserted blueprint blueprint_id = 8e153101-03cb-42e8-88f7-cb8007da28fb
15052025-04-30T06:42:43.831ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = BlueprintConfig
15062025-04-30T06:42:43.831ZTRACtest_snapshot_basic (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
15072025-04-30T06:42:43.831ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
15082025-04-30T06:42:43.831ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {} }
15092025-04-30T06:42:43.832ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
15102025-04-30T06:42:43.834ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "7b02ee83-4b89-4503-9f9b-0e66079c1bb7", "content-length": "169", "date": "Wed, 30 Apr 2025 06:42:42 GMT"} })
15112025-04-30T06:42:43.834ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/0/component/rot/caboose?firmware_slot=1
15122025-04-30T06:42:43.835ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
15132025-04-30T06:42:43.839ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = BlueprintConfig result = Ok(())
15142025-04-30T06:42:43.842ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "8a14c3a4-25c3-49d2-95e3-9b53e08fe033", "content-length": "169", "date": "Wed, 30 Apr 2025 06:42:42 GMT"} })
15152025-04-30T06:42:43.842ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/0/component/stage0/caboose?firmware_slot=0
15162025-04-30T06:42:43.843ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "2176fa05-2315-40cc-af86-cdcca0585b4f", "content-length": "178", "date": "Wed, 30 Apr 2025 06:42:42 GMT"} })
15172025-04-30T06:42:43.844ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/0/component/stage0/caboose?firmware_slot=1
15182025-04-30T06:42:43.845ZINFOtest_snapshot_basic (InternalApi): No networking records needed for clickhouse service
15192025-04-30T06:42:43.847ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "4625bf3d-2cc5-4bbf-ae60-17481e2db74b", "content-length": "178", "date": "Wed, 30 Apr 2025 06:42:42 GMT"} })
15202025-04-30T06:42:43.847ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/0/component/rot/cmpa
15212025-04-30T06:42:43.849ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "78412c9b-bed4-4971-bcba-b50e827f19eb", "content-length": "702", "date": "Wed, 30 Apr 2025 06:42:42 GMT"} })
15222025-04-30T06:42:43.852ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/0/component/rot/cfpa
15232025-04-30T06:42:43.852ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "f42ad2da-6d64-4fbc-956c-663e1c0dac86", "content-length": "727", "date": "Wed, 30 Apr 2025 06:42:42 GMT"} })
15242025-04-30T06:42:43.852ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/0/component/rot/cfpa
15252025-04-30T06:42:43.852ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "597e41bd-1c2c-47b4-a4e4-969a772c44f9", "content-length": "729", "date": "Wed, 30 Apr 2025 06:42:42 GMT"} })
15262025-04-30T06:42:43.852ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/0/component/rot/cfpa
15272025-04-30T06:42:43.854ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "8d6acf92-3427-41d6-afce-f783a7cbba0a", "content-length": "728", "date": "Wed, 30 Apr 2025 06:42:42 GMT"} })
15282025-04-30T06:42:43.854ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/1
15292025-04-30T06:42:43.855ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "161162d2-e424-4af6-a8e9-b3bb2df07b60", "content-length": "723", "date": "Wed, 30 Apr 2025 06:42:42 GMT"} })
15302025-04-30T06:42:43.856ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/1/component/sp/caboose?firmware_slot=0
15312025-04-30T06:42:43.858ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "69cb25cd-73f0-40c6-939c-45aef42dd23f", "content-length": "109", "date": "Wed, 30 Apr 2025 06:42:42 GMT"} })
15322025-04-30T06:42:43.858ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/1/component/sp/caboose?firmware_slot=1
15332025-04-30T06:42:43.861ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "beabba69-62f6-48a8-a992-0de723e199dd", "content-length": "109", "date": "Wed, 30 Apr 2025 06:42:42 GMT"} })
15342025-04-30T06:42:43.861ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/1/component/rot/caboose?firmware_slot=0
15352025-04-30T06:42:43.879ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "eba23527-6908-4ae4-ae0f-6ff187d303cb", "content-length": "169", "date": "Wed, 30 Apr 2025 06:42:43 GMT"} })
15362025-04-30T06:42:43.879ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/1/component/rot/caboose?firmware_slot=1
15372025-04-30T06:42:43.881ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "13d025b5-f401-4ba4-ba1b-4b9c1a16ae16", "content-length": "169", "date": "Wed, 30 Apr 2025 06:42:43 GMT"} })
15382025-04-30T06:42:43.883ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/1/component/stage0/caboose?firmware_slot=0
15392025-04-30T06:42:43.884ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "770f2b4d-9ff4-4c40-aa84-ef951219f9ae", "content-length": "178", "date": "Wed, 30 Apr 2025 06:42:43 GMT"} })
15402025-04-30T06:42:43.885ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/1/component/stage0/caboose?firmware_slot=1
15412025-04-30T06:42:43.889ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "b3afd960-fa41-4053-9e20-2aeb77a382c1", "content-length": "178", "date": "Wed, 30 Apr 2025 06:42:43 GMT"} })
15422025-04-30T06:42:43.889ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/1/component/rot/cmpa
15432025-04-30T06:42:43.891ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "77c91789-b415-4b88-8de9-eb13e5ec166a", "content-length": "702", "date": "Wed, 30 Apr 2025 06:42:43 GMT"} })
15442025-04-30T06:42:43.892ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/1/component/rot/cfpa
15452025-04-30T06:42:43.894ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "43da1445-3889-4d05-a5cd-23461d35c43e", "content-length": "727", "date": "Wed, 30 Apr 2025 06:42:43 GMT"} })
15462025-04-30T06:42:43.894ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/1/component/rot/cfpa
15472025-04-30T06:42:43.896ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "1b0069ad-6daf-417a-8a86-04423b4d7525", "content-length": "729", "date": "Wed, 30 Apr 2025 06:42:43 GMT"} })
15482025-04-30T06:42:43.896ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/1/component/rot/cfpa
15492025-04-30T06:42:43.897ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "3cd92c1f-48e7-4206-aa79-82e418057b65", "content-length": "728", "date": "Wed, 30 Apr 2025 06:42:43 GMT"} })
15502025-04-30T06:42:43.897ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
15512025-04-30T06:42:43.897ZTRACtest_snapshot_basic (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
15522025-04-30T06:42:43.897ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
15532025-04-30T06:42:43.897ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
15542025-04-30T06:42:43.898ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
15552025-04-30T06:42:43.899ZDEBGtest_snapshot_basic (service-firewall-rule-progator): resolved sleds for vpc oxide-services background_task = service_firewall_rule_propagation vpc_to_sled = []
15562025-04-30T06:42:43.899ZDEBGtest_snapshot_basic (service-firewall-rule-progator): sending firewall rules to sled agents background_task = service_firewall_rule_propagation
15572025-04-30T06:42:43.899ZINFOtest_snapshot_basic (service-firewall-rule-progator): updated firewall rules on 0 sleds background_task = service_firewall_rule_propagation
15582025-04-30T06:42:43.899ZDEBGtest_snapshot_basic (service-firewall-rule-progator): successfully propagated service firewall rules background_task = service_firewall_rule_propagation elapsed = 5.393368264s
15592025-04-30T06:42:43.899ZDEBGtest_snapshot_basic (BackgroundTasks): activation complete background_task = service_firewall_rule_propagation elapsed = 5.39342666s iteration = 1 status = {}
15602025-04-30T06:42:43.902ZINFOtest_snapshot_basic (InternalApi): Inserted networking records for nexus service
15612025-04-30T06:42:43.902ZINFOtest_snapshot_basic (InternalApi): No networking records needed for internal_dns service
15622025-04-30T06:42:43.902ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
15632025-04-30T06:42:43.908ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
15642025-04-30T06:42:43.908ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
15652025-04-30T06:42:43.908ZTRACtest_snapshot_basic (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
15662025-04-30T06:42:43.908ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
15672025-04-30T06:42:43.908ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
15682025-04-30T06:42:43.912ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
15692025-04-30T06:42:43.923ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
15702025-04-30T06:42:43.932ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
15712025-04-30T06:42:43.932ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
15722025-04-30T06:42:43.932ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
15732025-04-30T06:42:43.933ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
15742025-04-30T06:42:43.935ZINFOtest_snapshot_basic (InternalApi): Inserted networking records for external_dns service
15752025-04-30T06:42:43.935ZINFOtest_snapshot_basic (InternalApi): No networking records needed for crucible_pantry service
15762025-04-30T06:42:43.935ZINFOtest_snapshot_basic (InternalApi): No networking records needed for cockroach_db service
15772025-04-30T06:42:43.935ZINFOtest_snapshot_basic (InternalApi): Inserted service networking records
15782025-04-30T06:42:43.935ZINFOtest_snapshot_basic (InternalApi): Inserted physical disks
15792025-04-30T06:42:43.935ZINFOtest_snapshot_basic (InternalApi): Inserted zpools
15802025-04-30T06:42:43.935ZINFOtest_snapshot_basic (InternalApi): Inserted datasets
15812025-04-30T06:42:43.956ZINFOtest_snapshot_basic (InternalApi): Populated DNS tables for internal DNS
15822025-04-30T06:42:43.983ZINFOtest_snapshot_basic (InternalApi): Populated DNS tables for external DNS
15832025-04-30T06:42:43.984ZTRACtest_snapshot_basic (InternalApi): authorize begin action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Fleet
15842025-04-30T06:42:43.984ZTRACtest_snapshot_basic (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
15852025-04-30T06:42:43.984ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
15862025-04-30T06:42:43.984ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {} }
15872025-04-30T06:42:43.993ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
15882025-04-30T06:42:44.001ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
15892025-04-30T06:42:44.004ZDEBGtest_snapshot_basic (InternalApi): authorize result action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Fleet result = Ok(())
15902025-04-30T06:42:44.004ZTRACtest_snapshot_basic (InternalApi): authorize begin action = ModifyPolicy actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Fleet
15912025-04-30T06:42:44.004ZTRACtest_snapshot_basic (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
15922025-04-30T06:42:44.004ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
15932025-04-30T06:42:44.004ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {} }
15942025-04-30T06:42:44.004ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
15952025-04-30T06:42:44.013ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
15962025-04-30T06:42:44.020ZDEBGtest_snapshot_basic (InternalApi): authorize result action = ModifyPolicy actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Fleet result = Ok(())
15972025-04-30T06:42:44.051ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = DnsConfig
15982025-04-30T06:42:44.051ZTRACtest_snapshot_basic (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
15992025-04-30T06:42:44.051ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
16002025-04-30T06:42:44.051ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {} }
16012025-04-30T06:42:44.053ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
16022025-04-30T06:42:44.112ZDEBGtest_snapshot_basic (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:41923
16032025-04-30T06:42:44.113ZDEBGtest_snapshot_basic (BackgroundTasks): client request SledAgent = http://[::1]:41923 background_task = inventory_collection body = None method = GET uri = http://[::1]:41923/inventory
16042025-04-30T06:42:44.113ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
16052025-04-30T06:42:44.136ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = DnsConfig result = Ok(())
16062025-04-30T06:42:44.136ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = DnsConfig
16072025-04-30T06:42:44.136ZTRACtest_snapshot_basic (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
16082025-04-30T06:42:44.136ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
16092025-04-30T06:42:44.136ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {} }
16102025-04-30T06:42:44.137ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
16112025-04-30T06:42:44.137ZINFOtest_snapshot_basic (dropshot): accepted connection local_addr = [::1]:41923 remote_addr = [::1]:54950 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
16122025-04-30T06:42:44.141ZTRACtest_snapshot_basic (dropshot): incoming request local_addr = [::1]:41923 method = GET remote_addr = [::1]:54950 req_id = 26fcf5bc-c4df-48cb-9953-0527125fe071 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
16132025-04-30T06:42:44.144ZINFOtest_snapshot_basic (dropshot): request completed latency_us = 2937 local_addr = [::1]:41923 method = GET remote_addr = [::1]:54950 req_id = 26fcf5bc-c4df-48cb-9953-0527125fe071 response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
16142025-04-30T06:42:44.146ZDEBGtest_snapshot_basic (BackgroundTasks): client response SledAgent = http://[::1]:41923 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:41923/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "26fcf5bc-c4df-48cb-9953-0527125fe071", "content-length": "2769", "date": "Wed, 30 Apr 2025 06:42:43 GMT"} })
16152025-04-30T06:42:44.152ZDEBGtest_snapshot_basic (BackgroundTasks): begin collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
16162025-04-30T06:42:44.152ZDEBGtest_snapshot_basic (BackgroundTasks): end collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
16172025-04-30T06:42:44.152ZDEBGtest_snapshot_basic (BackgroundTasks): finished collection background_task = inventory_collection
16182025-04-30T06:42:44.157ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
16192025-04-30T06:42:44.157ZTRACtest_snapshot_basic (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
16202025-04-30T06:42:44.158ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
16212025-04-30T06:42:44.158ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
16222025-04-30T06:42:44.158ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
16232025-04-30T06:42:44.159ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
16242025-04-30T06:42:44.167ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = DnsConfig result = Ok(())
16252025-04-30T06:42:44.169ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
16262025-04-30T06:42:44.176ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
16272025-04-30T06:42:44.176ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
16282025-04-30T06:42:44.176ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
16292025-04-30T06:42:44.177ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
16302025-04-30T06:42:44.179ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = DnsConfig
16312025-04-30T06:42:44.179ZTRACtest_snapshot_basic (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
16322025-04-30T06:42:44.179ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
16332025-04-30T06:42:44.179ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {} }
16342025-04-30T06:42:44.180ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
16352025-04-30T06:42:44.184ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
16362025-04-30T06:42:44.194ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = DnsConfig result = Ok(())
16372025-04-30T06:42:44.204ZINFOtest_snapshot_basic (InternalApi): Created recovery silo
16382025-04-30T06:42:44.207ZINFOtest_snapshot_basic (InternalApi): Created recovery user
16392025-04-30T06:42:44.212ZINFOtest_snapshot_basic (InternalApi): Created recovery user's password
16402025-04-30T06:42:44.212ZTRACtest_snapshot_basic (InternalApi): authorize begin action = ModifyPolicy actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Silo { parent: Fleet, key: 22f1cf4f-1dcf-4dd7-821b-c632e4deed3c, lookup_type: ById(22f1cf4f-1dcf-4dd7-821b-c632e4deed3c) }
16412025-04-30T06:42:44.212ZTRACtest_snapshot_basic (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 22f1cf4f-1dcf-4dd7-821b-c632e4deed3c resource_type = Silo
16422025-04-30T06:42:44.212ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
16432025-04-30T06:42:44.212ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {} }
16442025-04-30T06:42:44.213ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
16452025-04-30T06:42:44.221ZTRACtest_snapshot_basic (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
16462025-04-30T06:42:44.222ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
16472025-04-30T06:42:44.222ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {} }
16482025-04-30T06:42:44.222ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
16492025-04-30T06:42:44.227ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
16502025-04-30T06:42:44.230ZDEBGtest_snapshot_basic (InternalApi): authorize result action = ModifyPolicy actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Silo { parent: Fleet, key: 22f1cf4f-1dcf-4dd7-821b-c632e4deed3c, lookup_type: ById(22f1cf4f-1dcf-4dd7-821b-c632e4deed3c) } result = Ok(())
16512025-04-30T06:42:44.230ZDEBGtest_snapshot_basic (InternalApi): Generated role assignment queries
16522025-04-30T06:42:44.241ZINFOtest_snapshot_basic (InternalApi): Granted Silo privileges
16532025-04-30T06:42:44.242ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Fleet
16542025-04-30T06:42:44.242ZTRACtest_snapshot_basic (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
16552025-04-30T06:42:44.242ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
16562025-04-30T06:42:44.242ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {} }
16572025-04-30T06:42:44.242ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
16582025-04-30T06:42:44.249ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
16592025-04-30T06:42:44.256ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Fleet result = Ok(())
16602025-04-30T06:42:44.291ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
16612025-04-30T06:42:44.291ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {} }
16622025-04-30T06:42:44.292ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
16632025-04-30T06:42:44.311ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
16642025-04-30T06:42:44.312ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {} }
16652025-04-30T06:42:44.313ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
16662025-04-30T06:42:44.323ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
16672025-04-30T06:42:44.323ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {} }
16682025-04-30T06:42:44.324ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
16692025-04-30T06:42:44.327ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ById(001de000-074c-4000-8000-000000000000) }
16702025-04-30T06:42:44.327ZTRACtest_snapshot_basic (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-4401-4000-8000-000000000000 resource_type = Project
16712025-04-30T06:42:44.327ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
16722025-04-30T06:42:44.327ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {} }
16732025-04-30T06:42:44.328ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
16742025-04-30T06:42:44.332ZTRACtest_snapshot_basic (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-5110-4000-8000-000000000001 resource_type = Silo
16752025-04-30T06:42:44.332ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
16762025-04-30T06:42:44.332ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {} }
16772025-04-30T06:42:44.332ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
16782025-04-30T06:42:44.336ZTRACtest_snapshot_basic (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
16792025-04-30T06:42:44.336ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
16802025-04-30T06:42:44.336ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {} }
16812025-04-30T06:42:44.338ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
16822025-04-30T06:42:44.345ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
16832025-04-30T06:42:44.353ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ById(001de000-074c-4000-8000-000000000000) } result = Ok(())
16842025-04-30T06:42:44.353ZTRACtest_snapshot_basic (InternalApi): successful lookup of siloed resource "Vpc" using built-in user
16852025-04-30T06:42:44.353ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ById(001de000-074c-4000-8000-000000000000) }
16862025-04-30T06:42:44.353ZTRACtest_snapshot_basic (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-4401-4000-8000-000000000000 resource_type = Project
16872025-04-30T06:42:44.353ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
16882025-04-30T06:42:44.353ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {} }
16892025-04-30T06:42:44.353ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
16902025-04-30T06:42:44.363ZTRACtest_snapshot_basic (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-5110-4000-8000-000000000001 resource_type = Silo
16912025-04-30T06:42:44.363ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
16922025-04-30T06:42:44.363ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {} }
16932025-04-30T06:42:44.364ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
16942025-04-30T06:42:44.368ZTRACtest_snapshot_basic (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
16952025-04-30T06:42:44.368ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
16962025-04-30T06:42:44.369ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {} }
16972025-04-30T06:42:44.372ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
16982025-04-30T06:42:44.382ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
16992025-04-30T06:42:44.390ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ById(001de000-074c-4000-8000-000000000000) } result = Ok(())
17002025-04-30T06:42:44.390ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
17012025-04-30T06:42:44.390ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {} }
17022025-04-30T06:42:44.390ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
17032025-04-30T06:42:44.400ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
17042025-04-30T06:42:44.400ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {} }
17052025-04-30T06:42:44.401ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
17062025-04-30T06:42:44.404ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
17072025-04-30T06:42:44.404ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {} }
17082025-04-30T06:42:44.404ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
17092025-04-30T06:42:44.408ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
17102025-04-30T06:42:44.408ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {} }
17112025-04-30T06:42:44.408ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
17122025-04-30T06:42:44.410ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ById(001de000-074c-4000-8000-000000000000) }
17132025-04-30T06:42:44.410ZTRACtest_snapshot_basic (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-4401-4000-8000-000000000000 resource_type = Project
17142025-04-30T06:42:44.410ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
17152025-04-30T06:42:44.410ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {} }
17162025-04-30T06:42:44.411ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
17172025-04-30T06:42:44.414ZTRACtest_snapshot_basic (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-5110-4000-8000-000000000001 resource_type = Silo
17182025-04-30T06:42:44.414ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
17192025-04-30T06:42:44.414ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {} }
17202025-04-30T06:42:44.415ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
17212025-04-30T06:42:44.418ZTRACtest_snapshot_basic (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
17222025-04-30T06:42:44.418ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
17232025-04-30T06:42:44.418ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {} }
17242025-04-30T06:42:44.419ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
17252025-04-30T06:42:44.421ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
17262025-04-30T06:42:44.429ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ById(001de000-074c-4000-8000-000000000000) } result = Ok(())
17272025-04-30T06:42:44.429ZTRACtest_snapshot_basic (InternalApi): successful lookup of siloed resource "Vpc" using built-in user
17282025-04-30T06:42:44.429ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
17292025-04-30T06:42:44.429ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {} }
17302025-04-30T06:42:44.429ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
17312025-04-30T06:42:44.432ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
17322025-04-30T06:42:44.432ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {} }
17332025-04-30T06:42:44.448ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
17342025-04-30T06:42:44.464ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
17352025-04-30T06:42:44.464ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {} }
17362025-04-30T06:42:44.466ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
17372025-04-30T06:42:44.470ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
17382025-04-30T06:42:44.470ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {} }
17392025-04-30T06:42:44.470ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
17402025-04-30T06:42:44.474ZTRACtest_snapshot_basic (InternalApi): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = VpcSubnet { parent: Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ByName("oxide-services") }, key: 001de000-c470-4000-8000-000000000001, lookup_type: ByName("external-dns") }
17412025-04-30T06:42:44.474ZTRACtest_snapshot_basic (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-4401-4000-8000-000000000000 resource_type = Project
17422025-04-30T06:42:44.474ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
17432025-04-30T06:42:44.474ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {} }
17442025-04-30T06:42:44.476ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
17452025-04-30T06:42:44.478ZTRACtest_snapshot_basic (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-5110-4000-8000-000000000001 resource_type = Silo
17462025-04-30T06:42:44.478ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
17472025-04-30T06:42:44.478ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {} }
17482025-04-30T06:42:44.479ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
17492025-04-30T06:42:44.482ZTRACtest_snapshot_basic (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
17502025-04-30T06:42:44.482ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
17512025-04-30T06:42:44.483ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {} }
17522025-04-30T06:42:44.483ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
17532025-04-30T06:42:44.487ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
17542025-04-30T06:42:44.494ZDEBGtest_snapshot_basic (InternalApi): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = VpcSubnet { parent: Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ByName("oxide-services") }, key: 001de000-c470-4000-8000-000000000001, lookup_type: ByName("external-dns") } result = Ok(())
17552025-04-30T06:42:44.494ZTRACtest_snapshot_basic (InternalApi): successful lookup of siloed resource "VpcSubnet" using built-in user
17562025-04-30T06:42:44.494ZTRACtest_snapshot_basic (InternalApi): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = VpcSubnet { parent: Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ByName("oxide-services") }, key: 001de000-c470-4000-8000-000000000001, lookup_type: ByName("external-dns") }
17572025-04-30T06:42:44.494ZTRACtest_snapshot_basic (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-4401-4000-8000-000000000000 resource_type = Project
17582025-04-30T06:42:44.494ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
17592025-04-30T06:42:44.494ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {} }
17602025-04-30T06:42:44.495ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
17612025-04-30T06:42:44.497ZTRACtest_snapshot_basic (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-5110-4000-8000-000000000001 resource_type = Silo
17622025-04-30T06:42:44.497ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
17632025-04-30T06:42:44.497ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {} }
17642025-04-30T06:42:44.499ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
17652025-04-30T06:42:44.503ZTRACtest_snapshot_basic (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
17662025-04-30T06:42:44.503ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
17672025-04-30T06:42:44.503ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {} }
17682025-04-30T06:42:44.504ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
17692025-04-30T06:42:44.507ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
17702025-04-30T06:42:44.515ZDEBGtest_snapshot_basic (InternalApi): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = VpcSubnet { parent: Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ByName("oxide-services") }, key: 001de000-c470-4000-8000-000000000001, lookup_type: ByName("external-dns") } result = Ok(())
17712025-04-30T06:42:44.515ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
17722025-04-30T06:42:44.515ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {} }
17732025-04-30T06:42:44.517ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
17742025-04-30T06:42:44.535ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
17752025-04-30T06:42:44.535ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {} }
17762025-04-30T06:42:44.535ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
17772025-04-30T06:42:44.548ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
17782025-04-30T06:42:44.548ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {} }
17792025-04-30T06:42:44.549ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
17802025-04-30T06:42:44.554ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
17812025-04-30T06:42:44.555ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {} }
17822025-04-30T06:42:44.555ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
17832025-04-30T06:42:44.564ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
17842025-04-30T06:42:44.564ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {} }
17852025-04-30T06:42:44.564ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
17862025-04-30T06:42:44.571ZTRACtest_snapshot_basic (InternalApi): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = VpcSubnet { parent: Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ByName("oxide-services") }, key: 001de000-c470-4000-8000-000000000002, lookup_type: ByName("nexus") }
17872025-04-30T06:42:44.571ZTRACtest_snapshot_basic (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-4401-4000-8000-000000000000 resource_type = Project
17882025-04-30T06:42:44.571ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
17892025-04-30T06:42:44.571ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {} }
17902025-04-30T06:42:44.571ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
17912025-04-30T06:42:44.585ZTRACtest_snapshot_basic (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-5110-4000-8000-000000000001 resource_type = Silo
17922025-04-30T06:42:44.585ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
17932025-04-30T06:42:44.585ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {} }
17942025-04-30T06:42:44.585ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
17952025-04-30T06:42:44.594ZTRACtest_snapshot_basic (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
17962025-04-30T06:42:44.594ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
17972025-04-30T06:42:44.594ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {} }
17982025-04-30T06:42:44.595ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
17992025-04-30T06:42:44.601ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
18002025-04-30T06:42:44.626ZDEBGtest_snapshot_basic (InternalApi): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = VpcSubnet { parent: Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ByName("oxide-services") }, key: 001de000-c470-4000-8000-000000000002, lookup_type: ByName("nexus") } result = Ok(())
18012025-04-30T06:42:44.626ZTRACtest_snapshot_basic (InternalApi): successful lookup of siloed resource "VpcSubnet" using built-in user
18022025-04-30T06:42:44.626ZTRACtest_snapshot_basic (InternalApi): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = VpcSubnet { parent: Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ByName("oxide-services") }, key: 001de000-c470-4000-8000-000000000002, lookup_type: ByName("nexus") }
18032025-04-30T06:42:44.626ZTRACtest_snapshot_basic (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-4401-4000-8000-000000000000 resource_type = Project
18042025-04-30T06:42:44.626ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
18052025-04-30T06:42:44.626ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {} }
18062025-04-30T06:42:44.627ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
18072025-04-30T06:42:44.633ZTRACtest_snapshot_basic (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-5110-4000-8000-000000000001 resource_type = Silo
18082025-04-30T06:42:44.633ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
18092025-04-30T06:42:44.633ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {} }
18102025-04-30T06:42:44.633ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
18112025-04-30T06:42:44.638ZTRACtest_snapshot_basic (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
18122025-04-30T06:42:44.638ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
18132025-04-30T06:42:44.638ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {} }
18142025-04-30T06:42:44.638ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
18152025-04-30T06:42:44.643ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
18162025-04-30T06:42:44.670ZDEBGtest_snapshot_basic (InternalApi): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = VpcSubnet { parent: Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ByName("oxide-services") }, key: 001de000-c470-4000-8000-000000000002, lookup_type: ByName("nexus") } result = Ok(())
18172025-04-30T06:42:44.670ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
18182025-04-30T06:42:44.670ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {} }
18192025-04-30T06:42:44.674ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
18202025-04-30T06:42:44.675ZINFOtest_snapshot_basic (BackgroundTasks): inserted inventory collection background_task = inventory_collection collection_id = d1ad44e5-18dd-4193-a70e-bf9042f0e0e1
18212025-04-30T06:42:44.675ZDEBGtest_snapshot_basic (BackgroundTasks): inventory collection complete background_task = inventory_collection collection_id = d1ad44e5-18dd-4193-a70e-bf9042f0e0e1 time_started = 2025-04-30 06:42:43.123811 UTC
18222025-04-30T06:42:44.676ZDEBGtest_snapshot_basic (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 5.282684204s iteration = 2 status = {"collection_id":"d1ad44e5-18dd-4193-a70e-bf9042f0e0e1","time_done":"2025-04-30 06:42:44.153050 UTC","time_started":"2025-04-30 06:42:43.123811 UTC"}
18232025-04-30T06:42:44.677ZDEBGtest_snapshot_basic (BackgroundTasks): activating background_task = blueprint_rendezvous iteration = 2 reason = Dependency
18242025-04-30T06:42:44.677ZWARNtest_snapshot_basic (BackgroundTasks): Blueprint rendezvous: skipped background_task = blueprint_rendezvous reason = no blueprint
18252025-04-30T06:42:44.677ZDEBGtest_snapshot_basic (BackgroundTasks): activation complete background_task = blueprint_rendezvous elapsed = 54.91\u{b5}s iteration = 2 status = {"error":"no blueprint"}
18262025-04-30T06:42:44.677ZDEBGtest_snapshot_basic (BackgroundTasks): activating background_task = physical_disk_adoption iteration = 2 reason = Dependency
18272025-04-30T06:42:44.677ZDEBGtest_snapshot_basic (BackgroundTasks): activation complete background_task = physical_disk_adoption elapsed = 10.3\u{b5}s iteration = 2 status = {"error":"task disabled"}
18282025-04-30T06:42:44.688ZDEBGtest_snapshot_basic (nexus): resolved names for firewall rules instance_interfaces = {} subnet_interfaces = {Name("external-dns"): [NetworkInterface { id: 94c5e059-ce01-475a-b1f0-c22bca5d9c16, kind: Service { id: 7e244b61-e4d7-4237-8aaa-92889da3453c }, name: Name("external-dns-7e244b61-e4d7-4237-8aaa-92889da3453c"), ip: 172.30.1.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 128, 0])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }], Name("nexus"): [NetworkInterface { id: cf6bcb96-a640-44ec-916e-463eed3d1741, kind: Service { id: 354a52d4-3078-4706-81ce-12c32f6bbeaa }, name: Name("nexus-354a52d4-3078-4706-81ce-12c32f6bbeaa"), ip: 172.30.2.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 128, 1])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }]} subnet_networks = {Name("external-dns"): [V4(Ipv4Network { addr: 172.30.1.0, prefix: 24 }), V6(Ipv6Network { addr: fd77:e9d2:9cd9:1::, prefix: 64 })], Name("nexus"): [V4(Ipv4Network { addr: 172.30.2.0, prefix: 24 }), V6(Ipv6Network { addr: fd77:e9d2:9cd9:2::, prefix: 64 })]} vpc_interfaces = {}
18292025-04-30T06:42:44.688ZTRACtest_snapshot_basic (InternalApi): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Fleet
18302025-04-30T06:42:44.688ZTRACtest_snapshot_basic (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
18312025-04-30T06:42:44.688ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
18322025-04-30T06:42:44.688ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {} }
18332025-04-30T06:42:44.690ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
18342025-04-30T06:42:44.692ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
18352025-04-30T06:42:44.696ZDEBGtest_snapshot_basic (InternalApi): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Fleet result = Ok(())
18362025-04-30T06:42:44.696ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
18372025-04-30T06:42:44.697ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {} }
18382025-04-30T06:42:44.697ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
18392025-04-30T06:42:44.701ZTRACtest_snapshot_basic (nexus): fetched allowlist from DB allowed = AllowList { id: 001de000-a110-4000-8000-000000000000, time_created: 2025-04-30T06:06:08.248500Z, time_modified: 2025-04-30T06:42:44.256494Z, allowed_ips: None }
18402025-04-30T06:42:44.701ZDEBGtest_snapshot_basic (nexus): Allowlist for user-facing services is set to allow any inbound traffic. Existing VPC firewall rules will not be modified.
18412025-04-30T06:42:44.701ZDEBGtest_snapshot_basic (nexus): resolved firewall rules for sled agents sled_agent_rules = [ResolvedVpcFirewallRule { action: Allow, direction: Inbound, filter_hosts: None, filter_ports: Some([L4PortRange("53")]), filter_protocols: Some([Udp]), priority: 65534, status: Enabled, targets: [NetworkInterface { id: 94c5e059-ce01-475a-b1f0-c22bca5d9c16, kind: Service { id: 7e244b61-e4d7-4237-8aaa-92889da3453c }, name: Name("external-dns-7e244b61-e4d7-4237-8aaa-92889da3453c"), ip: 172.30.1.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 128, 0])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }] }, ResolvedVpcFirewallRule { action: Allow, direction: Inbound, filter_hosts: None, filter_ports: Some([L4PortRange("80"), L4PortRange("443")]), filter_protocols: Some([Tcp]), priority: 65534, status: Enabled, targets: [NetworkInterface { id: cf6bcb96-a640-44ec-916e-463eed3d1741, kind: Service { id: 354a52d4-3078-4706-81ce-12c32f6bbeaa }, name: Name("nexus-354a52d4-3078-4706-81ce-12c32f6bbeaa"), ip: 172.30.2.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 128, 1])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }] }]
18422025-04-30T06:42:44.701ZDEBGtest_snapshot_basic (nexus): resolved 2 rules for sleds
18432025-04-30T06:42:44.805ZDEBGtest_snapshot_basic (nexus): resolved sleds for vpc oxide-services vpc_to_sled = [Sled { identity: SledIdentity { id: b6d65341-167c-41df-9b5c-41cded99c229, time_created: 2025-04-30T06:42:38.543918Z, time_modified: 2025-04-30T06:42:38.543918Z }, time_deleted: None, rcgen: Generation(Generation(1)), rack_id: c19a698f-c6f9-4a17-ae30-20d711b8f7dc, is_scrimlet: true, serial_number: "sim-b6d65341", part_number: "Unknown", revision: SqlU32(0), usable_hardware_threads: SqlU32(16), usable_physical_ram: ByteCount(ByteCount(34359738368)), reservoir_size: ByteCount(ByteCount(17179869184)), ip: ::1, port: SqlU16(41923), last_used_address: ::ffff, policy: InService, state: Active, sled_agent_gen: Generation(Generation(1)), repo_depot_port: SqlU16(46799) }]
18442025-04-30T06:42:44.805ZDEBGtest_snapshot_basic (nexus): sending firewall rules to sled agents
18452025-04-30T06:42:44.805ZTRACtest_snapshot_basic (InstanceAllocator): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000004ead, .. }) resource = Database
18462025-04-30T06:42:44.805ZDEBGtest_snapshot_basic (InstanceAllocator): roles roles = RoleSet { roles: {} }
18472025-04-30T06:42:44.806ZDEBGtest_snapshot_basic (InstanceAllocator): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000004ead, .. }) resource = Database result = Ok(())
18482025-04-30T06:42:44.812ZTRACtest_snapshot_basic (InstanceAllocator): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000004ead, .. }) resource = Sled { parent: Fleet, key: b6d65341-167c-41df-9b5c-41cded99c229, lookup_type: ById(b6d65341-167c-41df-9b5c-41cded99c229) }
18492025-04-30T06:42:44.812ZTRACtest_snapshot_basic (InstanceAllocator): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000004ead, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
18502025-04-30T06:42:44.812ZTRACtest_snapshot_basic (InstanceAllocator): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000004ead, .. }) resource = Database
18512025-04-30T06:42:44.812ZDEBGtest_snapshot_basic (InstanceAllocator): roles roles = RoleSet { roles: {} }
18522025-04-30T06:42:44.813ZDEBGtest_snapshot_basic (InstanceAllocator): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000004ead, .. }) resource = Database result = Ok(())
18532025-04-30T06:42:44.816ZDEBGtest_snapshot_basic (InstanceAllocator): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "viewer")} }
18542025-04-30T06:42:44.821ZDEBGtest_snapshot_basic (InstanceAllocator): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000004ead, .. }) resource = Sled { parent: Fleet, key: b6d65341-167c-41df-9b5c-41cded99c229, lookup_type: ById(b6d65341-167c-41df-9b5c-41cded99c229) } result = Ok(())
18552025-04-30T06:42:44.886ZDEBGtest_snapshot_basic (nexus): client request SledAgent = b6d65341-167c-41df-9b5c-41cded99c229 body = Some(Body) method = PUT uri = http://[::1]:41923/vpc/001de000-074c-4000-8000-000000000000/firewall/rules
18562025-04-30T06:42:44.887ZINFOtest_snapshot_basic (dropshot): accepted connection local_addr = [::1]:41923 remote_addr = [::1]:36437 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
18572025-04-30T06:42:44.888ZTRACtest_snapshot_basic (dropshot): incoming request local_addr = [::1]:41923 method = PUT remote_addr = [::1]:36437 req_id = 94d3cce8-ceb0-4d98-ba82-4db6c381aee1 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /vpc/001de000-074c-4000-8000-000000000000/firewall/rules
18582025-04-30T06:42:44.889ZINFOtest_snapshot_basic (dropshot): request completed latency_us = 1279 local_addr = [::1]:41923 method = PUT remote_addr = [::1]:36437 req_id = 94d3cce8-ceb0-4d98-ba82-4db6c381aee1 response_code = 204 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /vpc/001de000-074c-4000-8000-000000000000/firewall/rules
18592025-04-30T06:42:44.890ZDEBGtest_snapshot_basic (nexus): client response SledAgent = b6d65341-167c-41df-9b5c-41cded99c229 result = Ok(Response { url: "http://[::1]:41923/vpc/001de000-074c-4000-8000-000000000000/firewall/rules", status: 204, headers: {"x-request-id": "94d3cce8-ceb0-4d98-ba82-4db6c381aee1", "date": "Wed, 30 Apr 2025 06:42:44 GMT"} })
18602025-04-30T06:42:44.898ZINFOtest_snapshot_basic (nexus): updated firewall rules on 1 sleds
18612025-04-30T06:42:44.898ZTRACtest_snapshot_basic (ServiceBalancer): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-00000000bac3, .. }) resource = Database
18622025-04-30T06:42:44.898ZDEBGtest_snapshot_basic (ServiceBalancer): roles roles = RoleSet { roles: {} }
18632025-04-30T06:42:44.899ZDEBGtest_snapshot_basic (ServiceBalancer): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-00000000bac3, .. }) resource = Database result = Ok(())
18642025-04-30T06:42:44.899ZDEBGtest_snapshot_basic (BackgroundTasks): activating background_task = dns_config_internal dns_group = internal iteration = 2 reason = Signaled
18652025-04-30T06:42:44.899ZDEBGtest_snapshot_basic (BackgroundTasks): reading DNS version background_task = dns_config_internal dns_group = internal
18662025-04-30T06:42:44.899ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database
18672025-04-30T06:42:44.900ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
18682025-04-30T06:42:44.900ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
18692025-04-30T06:42:44.900ZDEBGtest_snapshot_basic (BackgroundTasks): activating background_task = dns_servers_internal dns_group = internal iteration = 2 reason = Signaled
18702025-04-30T06:42:44.900ZTRACtest_snapshot_basic (DnsResolver): lookup_all_socket_v6 srv dns_name = _nameservice._tcp.control-plane.oxide.internal
18712025-04-30T06:42:44.900ZDEBGtest_snapshot_basic (BackgroundTasks): activating background_task = dns_config_external dns_group = external iteration = 2 reason = Signaled
18722025-04-30T06:42:44.900ZDEBGtest_snapshot_basic (BackgroundTasks): reading DNS version background_task = dns_config_external dns_group = external
18732025-04-30T06:42:44.900ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database
18742025-04-30T06:42:44.900ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
18752025-04-30T06:42:44.901ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
18762025-04-30T06:42:44.901ZDEBGtest_snapshot_basic (BackgroundTasks): activating background_task = dns_servers_external dns_group = external iteration = 2 reason = Signaled
18772025-04-30T06:42:44.901ZTRACtest_snapshot_basic (DnsResolver): lookup_all_socket_v6 srv dns_name = _external-dns._tcp.control-plane.oxide.internal
18782025-04-30T06:42:44.901ZDEBGtest_snapshot_basic (BackgroundTasks): activating background_task = external_endpoints iteration = 2 reason = Signaled
18792025-04-30T06:42:44.901ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Fleet
18802025-04-30T06:42:44.901ZTRACtest_snapshot_basic (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = external_endpoints resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
18812025-04-30T06:42:44.901ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database
18822025-04-30T06:42:44.901ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
18832025-04-30T06:42:44.901ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
18842025-04-30T06:42:44.901ZDEBGtest_snapshot_basic (BackgroundTasks): activating background_task = inventory_collection iteration = 3 reason = Signaled
18852025-04-30T06:42:44.901ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
18862025-04-30T06:42:44.902ZTRACtest_snapshot_basic (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
18872025-04-30T06:42:44.902ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
18882025-04-30T06:42:44.902ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
18892025-04-30T06:42:44.902ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
18902025-04-30T06:42:44.903ZTRACtest_snapshot_basic (dns): buffer buffer = Length: 75 (0x4b) bytes\n0000: 19 f3 01 00 00 01 00 00 00 00 00 01 0c 5f 6e 61 ............._na\n0010: 6d 65 73 65 72 76 69 63 65 04 5f 74 63 70 0d 63 meservice._tcp.c\n0020: 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 ontrol-plane.oxi\n0030: 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 21 00 01 de.internal..!..\n0040: 00 00 29 04 d0 00 00 00 00 00 00 ..)........ kind = dns peer_addr = [::1]:53730 req_id = 1b1a348a-8190-4dd0-8915-79404acbc6ef
18912025-04-30T06:42:44.903ZDEBGtest_snapshot_basic (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 6643,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_nameservice._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_nameservice._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 12,\n 95,\n 110,\n 97,\n 109,\n 101,\n 115,\n 101,\n 114,\n 118,\n 105,\n 99,\n 101,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:53730 req_id = 1b1a348a-8190-4dd0-8915-79404acbc6ef
18922025-04-30T06:42:44.903ZDEBGtest_snapshot_basic (store): query key key = _nameservice._tcp
18932025-04-30T06:42:44.903ZDEBGtest_snapshot_basic (store): query key key = 6508a503-8e09-46a1-8cbe-d797e5d9a3aa.host
18942025-04-30T06:42:44.904ZDEBGtest_snapshot_basic (dns): dns response additional_records = [Record { name_labels: Name("6508a503-8e09-46a1-8cbe-d797e5d9a3aa.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:53730 query = LowerQuery { name: LowerName(Name("_nameservice._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_nameservice._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_nameservice._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 48482, target: Name("6508a503-8e09-46a1-8cbe-d797e5d9a3aa.host.control-plane.oxide.internal") })) }] req_id = 1b1a348a-8190-4dd0-8915-79404acbc6ef
18952025-04-30T06:42:44.904ZTRACtest_snapshot_basic (dns): buffer buffer = Length: 76 (0x4c) bytes\n0000: 91 8f 01 00 00 01 00 00 00 00 00 01 0d 5f 65 78 ............._ex\n0010: 74 65 72 6e 61 6c 2d 64 6e 73 04 5f 74 63 70 0d ternal-dns._tcp.\n0020: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0030: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 21 00 ide.internal..!.\n0040: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:64663 req_id = c14f7c27-3d19-48f8-9e23-7ceb9ed67f87
18962025-04-30T06:42:44.904ZDEBGtest_snapshot_basic (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 37263,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_external-dns._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_external-dns._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 13,\n 95,\n 101,\n 120,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 45,\n 100,\n 110,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:64663 req_id = c14f7c27-3d19-48f8-9e23-7ceb9ed67f87
18972025-04-30T06:42:44.904ZDEBGtest_snapshot_basic (store): query key key = _external-dns._tcp
18982025-04-30T06:42:44.906ZDEBGtest_snapshot_basic (store): query key key = 7e244b61-e4d7-4237-8aaa-92889da3453c.host
18992025-04-30T06:42:44.906ZDEBGtest_snapshot_basic (dns): dns response additional_records = [Record { name_labels: Name("7e244b61-e4d7-4237-8aaa-92889da3453c.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:64663 query = LowerQuery { name: LowerName(Name("_external-dns._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_external-dns._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_external-dns._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 46673, target: Name("7e244b61-e4d7-4237-8aaa-92889da3453c.host.control-plane.oxide.internal") })) }] req_id = c14f7c27-3d19-48f8-9e23-7ceb9ed67f87
19002025-04-30T06:42:44.907ZDEBGtest_snapshot_basic (DnsResolver): lookup_all_socket_v6 srv dns_name = _nameservice._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_nameservice._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_nameservice._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 48482, target: Name("6508a503-8e09-46a1-8cbe-d797e5d9a3aa.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("6508a503-8e09-46a1-8cbe-d797e5d9a3aa.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 4229, tv_nsec: 706322979 } })
19012025-04-30T06:42:44.907ZTRACtest_snapshot_basic (DnsResolver): lookup_service_targets: looking up SRV target
19022025-04-30T06:42:44.907ZDEBGtest_snapshot_basic (DnsResolver): lookup_all_socket_v6 srv dns_name = _external-dns._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_external-dns._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_external-dns._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 46673, target: Name("7e244b61-e4d7-4237-8aaa-92889da3453c.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("7e244b61-e4d7-4237-8aaa-92889da3453c.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 4229, tv_nsec: 706718317 } })
19032025-04-30T06:42:44.908ZTRACtest_snapshot_basic (DnsResolver): lookup_service_targets: looking up SRV target
19042025-04-30T06:42:44.908ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = DnsConfig
19052025-04-30T06:42:44.908ZTRACtest_snapshot_basic (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = dns_config_internal dns_group = internal resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
19062025-04-30T06:42:44.908ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database
19072025-04-30T06:42:44.908ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
19082025-04-30T06:42:44.908ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
19092025-04-30T06:42:44.909ZTRACtest_snapshot_basic (ServiceBalancer): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-00000000bac3, .. }) resource = Rack { parent: Fleet, key: c19a698f-c6f9-4a17-ae30-20d711b8f7dc, lookup_type: ById(c19a698f-c6f9-4a17-ae30-20d711b8f7dc) }
19102025-04-30T06:42:44.909ZTRACtest_snapshot_basic (ServiceBalancer): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-00000000bac3, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
19112025-04-30T06:42:44.909ZTRACtest_snapshot_basic (ServiceBalancer): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-00000000bac3, .. }) resource = Database
19122025-04-30T06:42:44.909ZDEBGtest_snapshot_basic (ServiceBalancer): roles roles = RoleSet { roles: {} }
19132025-04-30T06:42:44.910ZDEBGtest_snapshot_basic (ServiceBalancer): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-00000000bac3, .. }) resource = Database result = Ok(())
19142025-04-30T06:42:44.910ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = DnsConfig
19152025-04-30T06:42:44.910ZTRACtest_snapshot_basic (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = dns_config_external dns_group = external resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
19162025-04-30T06:42:44.910ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database
19172025-04-30T06:42:44.910ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
19182025-04-30T06:42:44.911ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
19192025-04-30T06:42:44.912ZTRACtest_snapshot_basic (dns): buffer buffer = Length: 99 (0x63) bytes\n0000: 1f a4 01 00 00 01 00 00 00 00 00 01 24 36 35 30 ............$650\n0010: 38 61 35 30 33 2d 38 65 30 39 2d 34 36 61 31 2d 8a503-8e09-46a1-\n0020: 38 63 62 65 2d 64 37 39 37 65 35 64 39 61 33 61 8cbe-d797e5d9a3a\n0030: 61 04 68 6f 73 74 0d 63 6f 6e 74 72 6f 6c 2d 70 a.host.control-p\n0040: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0050: 6e 61 6c 00 00 1c 00 01 00 00 29 04 d0 00 00 00 nal.......).....\n0060: 00 00 00 ... kind = dns peer_addr = [::1]:60461 req_id = c37a1483-7ac5-42e6-a69c-f133d9b54ae2
19202025-04-30T06:42:44.912ZDEBGtest_snapshot_basic (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 8100,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("6508a503-8e09-46a1-8cbe-d797e5d9a3aa.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("6508a503-8e09-46a1-8cbe-d797e5d9a3aa.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 36,\n 54,\n 53,\n 48,\n 56,\n 97,\n 53,\n 48,\n 51,\n 45,\n 56,\n 101,\n 48,\n 57,\n 45,\n 52,\n 54,\n 97,\n 49,\n 45,\n 56,\n 99,\n 98,\n 101,\n 45,\n 100,\n 55,\n 57,\n 55,\n 101,\n 53,\n 100,\n 57,\n 97,\n 51,\n 97,\n 97,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:60461 req_id = c37a1483-7ac5-42e6-a69c-f133d9b54ae2
19212025-04-30T06:42:44.912ZDEBGtest_snapshot_basic (store): query key key = 6508a503-8e09-46a1-8cbe-d797e5d9a3aa.host
19222025-04-30T06:42:44.912ZDEBGtest_snapshot_basic (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:60461 query = LowerQuery { name: LowerName(Name("6508a503-8e09-46a1-8cbe-d797e5d9a3aa.host.control-plane.oxide.internal.")), original: Query { name: Name("6508a503-8e09-46a1-8cbe-d797e5d9a3aa.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("6508a503-8e09-46a1-8cbe-d797e5d9a3aa.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = c37a1483-7ac5-42e6-a69c-f133d9b54ae2
19232025-04-30T06:42:44.912ZTRACtest_snapshot_basic (dns): buffer buffer = Length: 99 (0x63) bytes\n0000: e5 f8 01 00 00 01 00 00 00 00 00 01 24 37 65 32 ............$7e2\n0010: 34 34 62 36 31 2d 65 34 64 37 2d 34 32 33 37 2d 44b61-e4d7-4237-\n0020: 38 61 61 61 2d 39 32 38 38 39 64 61 33 34 35 33 8aaa-92889da3453\n0030: 63 04 68 6f 73 74 0d 63 6f 6e 74 72 6f 6c 2d 70 c.host.control-p\n0040: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0050: 6e 61 6c 00 00 1c 00 01 00 00 29 04 d0 00 00 00 nal.......).....\n0060: 00 00 00 ... kind = dns peer_addr = [::1]:50865 req_id = 3dfd1ab6-4009-475d-bebd-8d060a5586d7
19242025-04-30T06:42:44.912ZDEBGtest_snapshot_basic (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 58872,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("7e244b61-e4d7-4237-8aaa-92889da3453c.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("7e244b61-e4d7-4237-8aaa-92889da3453c.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 36,\n 55,\n 101,\n 50,\n 52,\n 52,\n 98,\n 54,\n 49,\n 45,\n 101,\n 52,\n 100,\n 55,\n 45,\n 52,\n 50,\n 51,\n 55,\n 45,\n 56,\n 97,\n 97,\n 97,\n 45,\n 57,\n 50,\n 56,\n 56,\n 57,\n 100,\n 97,\n 51,\n 52,\n 53,\n 51,\n 99,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:50865 req_id = 3dfd1ab6-4009-475d-bebd-8d060a5586d7
19252025-04-30T06:42:44.912ZDEBGtest_snapshot_basic (store): query key key = 7e244b61-e4d7-4237-8aaa-92889da3453c.host
19262025-04-30T06:42:44.912ZDEBGtest_snapshot_basic (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:50865 query = LowerQuery { name: LowerName(Name("7e244b61-e4d7-4237-8aaa-92889da3453c.host.control-plane.oxide.internal.")), original: Query { name: Name("7e244b61-e4d7-4237-8aaa-92889da3453c.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("7e244b61-e4d7-4237-8aaa-92889da3453c.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = 3dfd1ab6-4009-475d-bebd-8d060a5586d7
19272025-04-30T06:42:44.913ZINFOtest_snapshot_basic (BackgroundTasks): found DNS servers (initial) addresses = DnsServersList { addresses: [[::1]:48482] } background_task = dns_servers_internal dns_group = internal
19282025-04-30T06:42:44.914ZDEBGtest_snapshot_basic (BackgroundTasks): activation complete background_task = dns_servers_internal dns_group = internal elapsed = 13.472744ms iteration = 2 status = {"addresses":["[::1]:48482"]}
19292025-04-30T06:42:44.914ZINFOtest_snapshot_basic (BackgroundTasks): found DNS servers (initial) addresses = DnsServersList { addresses: [[::1]:46673] } background_task = dns_servers_external dns_group = external
19302025-04-30T06:42:44.914ZDEBGtest_snapshot_basic (BackgroundTasks): activation complete background_task = dns_servers_external dns_group = external elapsed = 13.027156ms iteration = 2 status = {"addresses":["[::1]:46673"]}
19312025-04-30T06:42:44.914ZDEBGtest_snapshot_basic (BackgroundTasks): activating background_task = dns_propagation_internal dns_group = internal iteration = 2 reason = Dependency
19322025-04-30T06:42:44.914ZWARNtest_snapshot_basic (BackgroundTasks): DNS propagation: skipped background_task = dns_propagation_internal dns_group = internal reason = no config
19332025-04-30T06:42:44.914ZDEBGtest_snapshot_basic (BackgroundTasks): activation complete background_task = dns_propagation_internal dns_group = internal elapsed = 24.158\u{b5}s iteration = 2 status = {"error":"no config"}
19342025-04-30T06:42:44.914ZDEBGtest_snapshot_basic (BackgroundTasks): activating background_task = dns_propagation_external dns_group = external iteration = 2 reason = Dependency
19352025-04-30T06:42:44.914ZWARNtest_snapshot_basic (BackgroundTasks): DNS propagation: skipped background_task = dns_propagation_external dns_group = external reason = no config
19362025-04-30T06:42:44.914ZDEBGtest_snapshot_basic (BackgroundTasks): activation complete background_task = dns_propagation_external dns_group = external elapsed = 10.311\u{b5}s iteration = 2 status = {"error":"no config"}
19372025-04-30T06:42:44.914ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
19382025-04-30T06:42:44.925ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Fleet result = Ok(())
19392025-04-30T06:42:44.925ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database
19402025-04-30T06:42:44.925ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
19412025-04-30T06:42:44.926ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
19422025-04-30T06:42:44.927ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
19432025-04-30T06:42:44.933ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
19442025-04-30T06:42:44.933ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
19452025-04-30T06:42:44.933ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
19462025-04-30T06:42:44.934ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
19472025-04-30T06:42:44.934ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
19482025-04-30T06:42:44.943ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = DnsConfig result = Ok(())
19492025-04-30T06:42:44.947ZDEBGtest_snapshot_basic (BackgroundTasks): found DNS version background_task = dns_config_internal dns_group = internal version = 1
19502025-04-30T06:42:44.947ZDEBGtest_snapshot_basic (BackgroundTasks): reading DNS config background_task = dns_config_internal dns_group = internal
19512025-04-30T06:42:44.947ZDEBGtest_snapshot_basic (BackgroundTasks): reading DNS zones background_task = dns_config_internal dns_group = internal
19522025-04-30T06:42:44.947ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = DnsConfig
19532025-04-30T06:42:44.947ZTRACtest_snapshot_basic (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = dns_config_internal dns_group = internal resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
19542025-04-30T06:42:44.947ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database
19552025-04-30T06:42:44.947ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
19562025-04-30T06:42:44.947ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
19572025-04-30T06:42:44.950ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
19582025-04-30T06:42:44.958ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = DnsConfig result = Ok(())
19592025-04-30T06:42:44.959ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = DnsConfig
19602025-04-30T06:42:44.959ZTRACtest_snapshot_basic (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = external_endpoints resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
19612025-04-30T06:42:44.959ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database
19622025-04-30T06:42:44.959ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
19632025-04-30T06:42:44.960ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
19642025-04-30T06:42:44.962ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
19652025-04-30T06:42:44.970ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = DnsConfig result = Ok(())
19662025-04-30T06:42:44.970ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database
19672025-04-30T06:42:44.970ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
19682025-04-30T06:42:44.971ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
19692025-04-30T06:42:44.972ZDEBGtest_snapshot_basic (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = [(d1ad44e5-18dd-4193-a70e-bf9042f0e0e1, 0)]
19702025-04-30T06:42:44.972ZTRACtest_snapshot_basic (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
19712025-04-30T06:42:44.973ZDEBGtest_snapshot_basic (ServiceBalancer): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
19722025-04-30T06:42:44.983ZDEBGtest_snapshot_basic (ServiceBalancer): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-00000000bac3, .. }) resource = Rack { parent: Fleet, key: c19a698f-c6f9-4a17-ae30-20d711b8f7dc, lookup_type: ById(c19a698f-c6f9-4a17-ae30-20d711b8f7dc) } result = Ok(())
19732025-04-30T06:42:44.991ZINFOtest_snapshot_basic (nexus): Rack initialized
19742025-04-30T06:42:44.992ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
19752025-04-30T06:42:44.992ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {} }
19762025-04-30T06:42:44.993ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
19772025-04-30T06:42:44.993ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
19782025-04-30T06:42:45.010ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = DnsConfig result = Ok(())
19792025-04-30T06:42:45.010ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database
19802025-04-30T06:42:45.011ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
19812025-04-30T06:42:45.012ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
19822025-04-30T06:42:45.014ZDEBGtest_snapshot_basic (BackgroundTasks): found zones background_task = dns_config_internal count = 1 dns_group = internal
19832025-04-30T06:42:45.014ZDEBGtest_snapshot_basic (BackgroundTasks): listing DNS names for zone background_task = dns_config_internal batch_size = 100 dns_group = internal dns_zone_id = 4f4253e1-1ed1-4c57-b631-b04d63cf1176 dns_zone_name = control-plane.oxide.internal found_so_far = 0 version = 1
19842025-04-30T06:42:45.014ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = DnsConfig
19852025-04-30T06:42:45.015ZTRACtest_snapshot_basic (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = dns_config_internal dns_group = internal resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
19862025-04-30T06:42:45.015ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database
19872025-04-30T06:42:45.015ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
19882025-04-30T06:42:45.016ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
19892025-04-30T06:42:45.016ZDEBGtest_snapshot_basic (BackgroundTasks): found DNS version background_task = dns_config_external dns_group = external version = 2
19902025-04-30T06:42:45.016ZDEBGtest_snapshot_basic (BackgroundTasks): reading DNS config background_task = dns_config_external dns_group = external
19912025-04-30T06:42:45.016ZDEBGtest_snapshot_basic (BackgroundTasks): reading DNS zones background_task = dns_config_external dns_group = external
19922025-04-30T06:42:45.016ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = DnsConfig
19932025-04-30T06:42:45.017ZTRACtest_snapshot_basic (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = dns_config_external dns_group = external resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
19942025-04-30T06:42:45.017ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database
19952025-04-30T06:42:45.017ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
19962025-04-30T06:42:45.018ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
19972025-04-30T06:42:45.020ZTRACtest_snapshot_basic (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: c1 60 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 .`..........._mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:52273 req_id = cd7e38ac-4234-4cd1-8aa0-6fa282f44632
19982025-04-30T06:42:45.020ZDEBGtest_snapshot_basic (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 49504,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:52273 req_id = cd7e38ac-4234-4cd1-8aa0-6fa282f44632
19992025-04-30T06:42:45.020ZDEBGtest_snapshot_basic (store): query key key = _mgs._tcp
20002025-04-30T06:42:45.020ZDEBGtest_snapshot_basic (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
20012025-04-30T06:42:45.020ZDEBGtest_snapshot_basic (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:52273 query = LowerQuery { name: LowerName(Name("_mgs._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 41502, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = cd7e38ac-4234-4cd1-8aa0-6fa282f44632
20022025-04-30T06:42:45.022ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
20032025-04-30T06:42:45.030ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = DnsConfig result = Ok(())
20042025-04-30T06:42:45.030ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database
20052025-04-30T06:42:45.030ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
20062025-04-30T06:42:45.031ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
20072025-04-30T06:42:45.035ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
20082025-04-30T06:42:45.045ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = DnsConfig result = Ok(())
20092025-04-30T06:42:45.045ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database
20102025-04-30T06:42:45.045ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
20112025-04-30T06:42:45.045ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
20122025-04-30T06:42:45.048ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
20132025-04-30T06:42:45.048ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {} }
20142025-04-30T06:42:45.049ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
20152025-04-30T06:42:45.049ZDEBGtest_snapshot_basic (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 41502, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 4229, tv_nsec: 848559616 } })
20162025-04-30T06:42:45.049ZTRACtest_snapshot_basic (DnsResolver): lookup_service_targets: looking up SRV target
20172025-04-30T06:42:45.050ZDEBGtest_snapshot_basic (BackgroundTasks): found all DNS names for zone background_task = dns_config_internal dns_group = internal dns_zone_id = 4f4253e1-1ed1-4c57-b631-b04d63cf1176 dns_zone_name = control-plane.oxide.internal found_so_far = 21 version = 1
20182025-04-30T06:42:45.050ZDEBGtest_snapshot_basic (BackgroundTasks): read DNS config background_task = dns_config_internal dns_group = internal nzones = 1 version = 1
20192025-04-30T06:42:45.050ZINFOtest_snapshot_basic (BackgroundTasks): found latest generation (first find) background_task = dns_config_internal dns_group = internal generation = 1
20202025-04-30T06:42:45.050ZDEBGtest_snapshot_basic (BackgroundTasks): activation complete background_task = dns_config_internal dns_group = internal elapsed = 150.486064ms iteration = 2 status = {"generation":1}
20212025-04-30T06:42:45.050ZDEBGtest_snapshot_basic (BackgroundTasks): activating background_task = dns_propagation_internal dns_group = internal iteration = 3 reason = Dependency
20222025-04-30T06:42:45.115ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = dns_propagation_internal body = Some(Body) dns_group = internal dns_server_url = http://[::1]:48482 generation = 1 method = PUT servers = DnsServersList { addresses: [[::1]:48482] } uri = http://[::1]:48482/config
20232025-04-30T06:42:45.116ZDEBGtest_snapshot_basic (BackgroundTasks): found zones background_task = dns_config_external count = 1 dns_group = external
20242025-04-30T06:42:45.116ZDEBGtest_snapshot_basic (BackgroundTasks): listing DNS names for zone background_task = dns_config_external batch_size = 100 dns_group = external dns_zone_id = 0de4a429-d47b-4be9-9219-c9f8474a3874 dns_zone_name = oxide-dev.test found_so_far = 0 version = 2
20252025-04-30T06:42:45.116ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = DnsConfig
20262025-04-30T06:42:45.116ZTRACtest_snapshot_basic (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = dns_config_external dns_group = external resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
20272025-04-30T06:42:45.116ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database
20282025-04-30T06:42:45.116ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
20292025-04-30T06:42:45.117ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
20302025-04-30T06:42:45.118ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Fleet
20312025-04-30T06:42:45.118ZTRACtest_snapshot_basic (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = external_endpoints resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
20322025-04-30T06:42:45.118ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database
20332025-04-30T06:42:45.118ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
20342025-04-30T06:42:45.119ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
20352025-04-30T06:42:45.120ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
20362025-04-30T06:42:45.128ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = DnsConfig result = Ok(())
20372025-04-30T06:42:45.128ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database
20382025-04-30T06:42:45.128ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
20392025-04-30T06:42:45.128ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
20402025-04-30T06:42:45.129ZINFOtest_snapshot_basic (http): accepted connection kind = dns local_addr = [::1]:48482 remote_addr = [::1]:56170
20412025-04-30T06:42:45.130ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
20422025-04-30T06:42:45.134ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Fleet result = Ok(())
20432025-04-30T06:42:45.134ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database
20442025-04-30T06:42:45.134ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
20452025-04-30T06:42:45.134ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
20462025-04-30T06:42:45.135ZDEBGtest_snapshot_basic (BackgroundTasks): found all DNS names for zone background_task = dns_config_external dns_group = external dns_zone_id = 0de4a429-d47b-4be9-9219-c9f8474a3874 dns_zone_name = oxide-dev.test found_so_far = 1 version = 2
20472025-04-30T06:42:45.135ZDEBGtest_snapshot_basic (BackgroundTasks): read DNS config background_task = dns_config_external dns_group = external nzones = 1 version = 2
20482025-04-30T06:42:45.135ZINFOtest_snapshot_basic (BackgroundTasks): found latest generation (first find) background_task = dns_config_external dns_group = external generation = 2
20492025-04-30T06:42:45.135ZDEBGtest_snapshot_basic (BackgroundTasks): activation complete background_task = dns_config_external dns_group = external elapsed = 234.663873ms iteration = 2 status = {"generation":2}
20502025-04-30T06:42:45.135ZDEBGtest_snapshot_basic (BackgroundTasks): activating background_task = dns_propagation_external dns_group = external iteration = 3 reason = Dependency
20512025-04-30T06:42:45.187ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = dns_propagation_external body = Some(Body) dns_group = external dns_server_url = http://[::1]:46673 generation = 2 method = PUT servers = DnsServersList { addresses: [[::1]:46673] } uri = http://[::1]:46673/config
20522025-04-30T06:42:45.191ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
20532025-04-30T06:42:45.191ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {} }
20542025-04-30T06:42:45.192ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
20552025-04-30T06:42:45.192ZINFOtest_snapshot_basic (http): accepted connection kind = dns local_addr = [::1]:46673 remote_addr = [::1]:34098
20562025-04-30T06:42:45.192ZTRACtest_snapshot_basic (http): incoming request kind = dns local_addr = [::1]:48482 method = PUT remote_addr = [::1]:56170 req_id = 7f16d06b-813b-4b71-a3fe-7abe68b247f0 uri = /config
20572025-04-30T06:42:45.194ZTRACtest_snapshot_basic (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: b5 1f 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e ............-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:53652 req_id = 6b260362-bdf0-4d37-a49a-403e455b5d70
20582025-04-30T06:42:45.194ZDEBGtest_snapshot_basic (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 46367,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:53652 req_id = 6b260362-bdf0-4d37-a49a-403e455b5d70
20592025-04-30T06:42:45.194ZDEBGtest_snapshot_basic (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
20602025-04-30T06:42:45.194ZDEBGtest_snapshot_basic (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:53652 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = 6b260362-bdf0-4d37-a49a-403e455b5d70
20612025-04-30T06:42:45.195ZINFOtest_snapshot_basic (store): attempting generation update new_generation = 1 req_id = 7f16d06b-813b-4b71-a3fe-7abe68b247f0
20622025-04-30T06:42:45.195ZINFOtest_snapshot_basic (store): updated generation new_generation = 1 req_id = 7f16d06b-813b-4b71-a3fe-7abe68b247f0
20632025-04-30T06:42:45.195ZINFOtest_snapshot_basic (http): request completed kind = dns latency_us = 2910 local_addr = [::1]:48482 method = PUT remote_addr = [::1]:56170 req_id = 7f16d06b-813b-4b71-a3fe-7abe68b247f0 response_code = 204 uri = /config
20642025-04-30T06:42:45.308ZTRACtest_snapshot_basic (DnsResolver): lookup_all_socket_v6 srv dns_name = _clickhouse-admin-keeper._tcp.control-plane.oxide.internal
20652025-04-30T06:42:45.309ZTRACtest_snapshot_basic (http): incoming request kind = dns local_addr = [::1]:46673 method = PUT remote_addr = [::1]:34098 req_id = 5a9b8508-3bd6-42b7-b370-f83da5900365 uri = /config
20662025-04-30T06:42:45.309ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[::1]:48482 generation = 1 result = Ok(Response { url: "http://[::1]:48482/config", status: 204, headers: {"x-request-id": "7f16d06b-813b-4b71-a3fe-7abe68b247f0", "date": "Wed, 30 Apr 2025 06:42:44 GMT"} }) servers = DnsServersList { addresses: [[::1]:48482] }
20672025-04-30T06:42:45.309ZINFOtest_snapshot_basic (BackgroundTasks): DNS server now at generation 1 background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[::1]:48482 generation = 1 servers = DnsServersList { addresses: [[::1]:48482] }
20682025-04-30T06:42:45.311ZINFOtest_snapshot_basic (BackgroundTasks): DNS propagation: done background_task = dns_propagation_internal dns_group = internal generation = 1 servers = DnsServersList { addresses: [[::1]:48482] }
20692025-04-30T06:42:45.311ZDEBGtest_snapshot_basic (BackgroundTasks): activation complete background_task = dns_propagation_internal dns_group = internal elapsed = 260.736881ms iteration = 3 status = {"generation":1,"server_results":{"[::1]:48482":{"Ok":null}}}
20702025-04-30T06:42:45.311ZINFOtest_snapshot_basic (store): attempting generation update new_generation = 2 req_id = 5a9b8508-3bd6-42b7-b370-f83da5900365
20712025-04-30T06:42:45.311ZINFOtest_snapshot_basic (store): pruning trees for generations newer than 0
20722025-04-30T06:42:45.311ZDEBGtest_snapshot_basic (store): creating tree tree_name = generation_2_zone_oxide-dev.test
20732025-04-30T06:42:45.311ZDEBGtest_snapshot_basic (store): updating current config
20742025-04-30T06:42:45.311ZDEBGtest_snapshot_basic (store): flushing default tree
20752025-04-30T06:42:45.311ZINFOtest_snapshot_basic (store): pruning trees for generations older than 2 keep = 3
20762025-04-30T06:42:45.312ZINFOtest_snapshot_basic (store): updated generation new_generation = 2 req_id = 5a9b8508-3bd6-42b7-b370-f83da5900365
20772025-04-30T06:42:45.312ZINFOtest_snapshot_basic (http): request completed kind = dns latency_us = 3113 local_addr = [::1]:46673 method = PUT remote_addr = [::1]:34098 req_id = 5a9b8508-3bd6-42b7-b370-f83da5900365 response_code = 204 uri = /config
20782025-04-30T06:42:45.312ZINFOtest_snapshot_basic (BackgroundTasks): found Silo/DNS/TLS config (initial) background_task = external_endpoints config = ExternalEndpoints { by_dns_name: {"default-silo.sys.oxide-dev.test": ExternalEndpoint { silo_id: 001de000-5110-4000-8000-000000000000, db_silo: Silo { identity: SiloIdentity { id: 001de000-5110-4000-8000-000000000000, name: Name(Name("default-silo")), description: "default silo", time_created: 2025-04-30T06:42:36.859865Z, time_modified: 2025-04-30T06:42:36.859865Z, time_deleted: None }, discoverable: false, authentication_mode: Local, user_provision_type: ApiOnly, mapped_fleet_roles: Object {}, rcgen: Generation(Generation(1)) }, tls_certs: [] }, "test-suite-silo.sys.oxide-dev.test": ExternalEndpoint { silo_id: 22f1cf4f-1dcf-4dd7-821b-c632e4deed3c, db_silo: Silo { identity: SiloIdentity { id: 22f1cf4f-1dcf-4dd7-821b-c632e4deed3c, name: Name(Name("test-suite-silo")), description: "built-in recovery Silo", time_created: 2025-04-30T06:42:43.984146Z, time_modified: 2025-04-30T06:42:43.984146Z, time_deleted: None }, discoverable: false, authentication_mode: Local, user_provision_type: ApiOnly, mapped_fleet_roles: Object {"admin": Array [String("admin")]}, rcgen: Generation(Generation(1)) }, tls_certs: [] }}, warnings: [NoSiloCerts { silo_id: 001de000-5110-4000-8000-000000000000, dns_name: "default-silo.sys.oxide-dev.test" }, NoSiloCerts { silo_id: 22f1cf4f-1dcf-4dd7-821b-c632e4deed3c, dns_name: "test-suite-silo.sys.oxide-dev.test" }], default_endpoint: Some(ExternalEndpoint { silo_id: 22f1cf4f-1dcf-4dd7-821b-c632e4deed3c, db_silo: Silo { identity: SiloIdentity { id: 22f1cf4f-1dcf-4dd7-821b-c632e4deed3c, name: Name(Name("test-suite-silo")), description: "built-in recovery Silo", time_created: 2025-04-30T06:42:43.984146Z, time_modified: 2025-04-30T06:42:43.984146Z, time_deleted: None }, discoverable: false, authentication_mode: Local, user_provision_type: ApiOnly, mapped_fleet_roles: Object {"admin": Array [String("admin")]}, rcgen: Generation(Generation(1)) }, tls_certs: [] }) }
20792025-04-30T06:42:45.313ZDEBGtest_snapshot_basic (BackgroundTasks): activation complete background_task = external_endpoints elapsed = 411.631299ms iteration = 2 status = {"by_dns_name":{"default-silo.sys.oxide-dev.test":{"silo_id":"001de000-5110-4000-8000-000000000000","tls_certs":[]},"test-suite-silo.sys.oxide-dev.test":{"silo_id":"22f1cf4f-1dcf-4dd7-821b-c632e4deed3c","tls_certs":[]}},"default_endpoint":{"silo_id":"22f1cf4f-1dcf-4dd7-821b-c632e4deed3c","tls_certs":[]},"warnings":["silo 001de000-5110-4000-8000-000000000000 with DNS name \\"default-silo.sys.oxide-dev.test\\" has no usable certificates","silo 22f1cf4f-1dcf-4dd7-821b-c632e4deed3c with DNS name \\"test-suite-silo.sys.oxide-dev.test\\" has no usable certificates"]}
20802025-04-30T06:42:45.314ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = dns_propagation_external dns_group = external dns_server_url = http://[::1]:46673 generation = 2 result = Ok(Response { url: "http://[::1]:46673/config", status: 204, headers: {"x-request-id": "5a9b8508-3bd6-42b7-b370-f83da5900365", "date": "Wed, 30 Apr 2025 06:42:44 GMT"} }) servers = DnsServersList { addresses: [[::1]:46673] }
20812025-04-30T06:42:45.314ZINFOtest_snapshot_basic (BackgroundTasks): DNS server now at generation 2 background_task = dns_propagation_external dns_group = external dns_server_url = http://[::1]:46673 generation = 2 servers = DnsServersList { addresses: [[::1]:46673] }
20822025-04-30T06:42:45.317ZINFOtest_snapshot_basic (BackgroundTasks): DNS propagation: done background_task = dns_propagation_external dns_group = external generation = 2 servers = DnsServersList { addresses: [[::1]:46673] }
20832025-04-30T06:42:45.317ZDEBGtest_snapshot_basic (BackgroundTasks): activation complete background_task = dns_propagation_external dns_group = external elapsed = 181.889574ms iteration = 3 status = {"generation":2,"server_results":{"[::1]:46673":{"Ok":null}}}
20842025-04-30T06:42:45.317ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ById(001de000-074c-4000-8000-000000000000) }
20852025-04-30T06:42:45.317ZTRACtest_snapshot_basic (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-4401-4000-8000-000000000000 resource_type = Project
20862025-04-30T06:42:45.317ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
20872025-04-30T06:42:45.317ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {} }
20882025-04-30T06:42:45.318ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
20892025-04-30T06:42:45.320ZTRACtest_snapshot_basic (dns): buffer buffer = Length: 87 (0x57) bytes\n0000: 22 eb 01 00 00 01 00 00 00 00 00 01 18 5f 63 6c "............_cl\n0010: 69 63 6b 68 6f 75 73 65 2d 61 64 6d 69 6e 2d 6b ickhouse-admin-k\n0020: 65 65 70 65 72 04 5f 74 63 70 0d 63 6f 6e 74 72 eeper._tcp.contr\n0030: 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 69 ol-plane.oxide.i\n0040: 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 04 nternal..!....).\n0050: d0 00 00 00 00 00 00 ....... kind = dns peer_addr = [::1]:57851 req_id = beb1bcfd-79e8-4d18-9f40-dc09c20e4cab
20902025-04-30T06:42:45.320ZDEBGtest_snapshot_basic (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 8939,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 24,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 97,\n 100,\n 109,\n 105,\n 110,\n 45,\n 107,\n 101,\n 101,\n 112,\n 101,\n 114,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:57851 req_id = beb1bcfd-79e8-4d18-9f40-dc09c20e4cab
20912025-04-30T06:42:45.320ZDEBGtest_snapshot_basic (store): query key key = _clickhouse-admin-keeper._tcp
20922025-04-30T06:42:45.320ZERROtest_snapshot_basic (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 8939, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } NXDOMAIN: no records found for name: "_clickhouse-admin-keeper._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:57851 req_id = beb1bcfd-79e8-4d18-9f40-dc09c20e4cab
20932025-04-30T06:42:45.321ZDEBGtest_snapshot_basic (BackgroundTasks): begin collection background_task = inventory_collection
20942025-04-30T06:42:45.321ZDEBGtest_snapshot_basic (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:41502
20952025-04-30T06:42:45.321ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/ignition
20962025-04-30T06:42:45.323ZTRACtest_snapshot_basic (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-5110-4000-8000-000000000001 resource_type = Silo
20972025-04-30T06:42:45.323ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
20982025-04-30T06:42:45.323ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {} }
20992025-04-30T06:42:45.325ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
21002025-04-30T06:42:45.338ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "536d1c47-5daa-4eb2-acee-4edd2b86c632", "content-length": "835", "date": "Wed, 30 Apr 2025 06:42:44 GMT"} })
21012025-04-30T06:42:45.339ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/0
21022025-04-30T06:42:45.339ZTRACtest_snapshot_basic (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
21032025-04-30T06:42:45.339ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
21042025-04-30T06:42:45.339ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {} }
21052025-04-30T06:42:45.340ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
21062025-04-30T06:42:45.343ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "125ae253-cc92-47c1-b1c2-8a9923c1aa4e", "content-length": "734", "date": "Wed, 30 Apr 2025 06:42:44 GMT"} })
21072025-04-30T06:42:45.343ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/0/component/sp/caboose?firmware_slot=0
21082025-04-30T06:42:45.344ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
21092025-04-30T06:42:45.351ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ById(001de000-074c-4000-8000-000000000000) } result = Ok(())
21102025-04-30T06:42:45.351ZTRACtest_snapshot_basic (InternalApi): successful lookup of siloed resource "Vpc" using built-in user
21112025-04-30T06:42:45.351ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ById(001de000-074c-4000-8000-000000000000) }
21122025-04-30T06:42:45.351ZTRACtest_snapshot_basic (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-4401-4000-8000-000000000000 resource_type = Project
21132025-04-30T06:42:45.351ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
21142025-04-30T06:42:45.352ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {} }
21152025-04-30T06:42:45.352ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
21162025-04-30T06:42:45.354ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "d22a4c1c-18e3-4ed1-9b2a-859d526fc7fd", "content-length": "111", "date": "Wed, 30 Apr 2025 06:42:44 GMT"} })
21172025-04-30T06:42:45.355ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/0/component/sp/caboose?firmware_slot=1
21182025-04-30T06:42:45.355ZTRACtest_snapshot_basic (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-5110-4000-8000-000000000001 resource_type = Silo
21192025-04-30T06:42:45.355ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
21202025-04-30T06:42:45.355ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {} }
21212025-04-30T06:42:45.356ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
21222025-04-30T06:42:45.359ZTRACtest_snapshot_basic (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
21232025-04-30T06:42:45.360ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
21242025-04-30T06:42:45.360ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {} }
21252025-04-30T06:42:45.361ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
21262025-04-30T06:42:45.363ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "b4e7964f-131f-4fc8-ae19-b86faa50d703", "content-length": "111", "date": "Wed, 30 Apr 2025 06:42:44 GMT"} })
21272025-04-30T06:42:45.363ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/0/component/rot/caboose?firmware_slot=0
21282025-04-30T06:42:45.364ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
21292025-04-30T06:42:45.374ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ById(001de000-074c-4000-8000-000000000000) } result = Ok(())
21302025-04-30T06:42:45.374ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
21312025-04-30T06:42:45.374ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {} }
21322025-04-30T06:42:45.374ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
21332025-04-30T06:42:45.379ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
21342025-04-30T06:42:45.379ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {} }
21352025-04-30T06:42:45.381ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
21362025-04-30T06:42:45.384ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "43da3a15-5efb-4e0d-9185-0e6d47533047", "content-length": "170", "date": "Wed, 30 Apr 2025 06:42:44 GMT"} })
21372025-04-30T06:42:45.385ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/0/component/rot/caboose?firmware_slot=1
21382025-04-30T06:42:45.385ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
21392025-04-30T06:42:45.385ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {} }
21402025-04-30T06:42:45.387ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
21412025-04-30T06:42:45.390ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
21422025-04-30T06:42:45.390ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {} }
21432025-04-30T06:42:45.391ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
21442025-04-30T06:42:45.392ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "81c181df-9471-4703-bf34-98dca58eac31", "content-length": "170", "date": "Wed, 30 Apr 2025 06:42:44 GMT"} })
21452025-04-30T06:42:45.392ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/0/component/stage0/caboose?firmware_slot=0
21462025-04-30T06:42:45.393ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ById(001de000-074c-4000-8000-000000000000) }
21472025-04-30T06:42:45.393ZTRACtest_snapshot_basic (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-4401-4000-8000-000000000000 resource_type = Project
21482025-04-30T06:42:45.393ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
21492025-04-30T06:42:45.393ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {} }
21502025-04-30T06:42:45.394ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
21512025-04-30T06:42:45.396ZTRACtest_snapshot_basic (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-5110-4000-8000-000000000001 resource_type = Silo
21522025-04-30T06:42:45.396ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
21532025-04-30T06:42:45.396ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {} }
21542025-04-30T06:42:45.398ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
21552025-04-30T06:42:45.400ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "96c6ee88-ef6c-421f-9de8-ad9e1ce5ac9e", "content-length": "179", "date": "Wed, 30 Apr 2025 06:42:44 GMT"} })
21562025-04-30T06:42:45.400ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/0/component/stage0/caboose?firmware_slot=1
21572025-04-30T06:42:45.401ZTRACtest_snapshot_basic (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
21582025-04-30T06:42:45.401ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
21592025-04-30T06:42:45.401ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {} }
21602025-04-30T06:42:45.401ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
21612025-04-30T06:42:45.404ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "37832668-0ce0-4bcd-bbb8-4829314d3d51", "content-length": "179", "date": "Wed, 30 Apr 2025 06:42:44 GMT"} })
21622025-04-30T06:42:45.404ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/0/component/rot/cmpa
21632025-04-30T06:42:45.405ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
21642025-04-30T06:42:45.417ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ById(001de000-074c-4000-8000-000000000000) } result = Ok(())
21652025-04-30T06:42:45.417ZTRACtest_snapshot_basic (InternalApi): successful lookup of siloed resource "Vpc" using built-in user
21662025-04-30T06:42:45.417ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
21672025-04-30T06:42:45.417ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {} }
21682025-04-30T06:42:45.418ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
21692025-04-30T06:42:45.419ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "c3135435-d9e7-41a5-9209-af180b7e5224", "content-length": "702", "date": "Wed, 30 Apr 2025 06:42:44 GMT"} })
21702025-04-30T06:42:45.419ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/0/component/rot/cfpa
21712025-04-30T06:42:45.422ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "c053cbce-2e07-400b-afd1-889bc3cc07d1", "content-length": "727", "date": "Wed, 30 Apr 2025 06:42:44 GMT"} })
21722025-04-30T06:42:45.422ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/0/component/rot/cfpa
21732025-04-30T06:42:45.422ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
21742025-04-30T06:42:45.422ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {} }
21752025-04-30T06:42:45.422ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
21762025-04-30T06:42:45.423ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "e95e353f-6ae3-48b6-8d53-28394ce90acb", "content-length": "729", "date": "Wed, 30 Apr 2025 06:42:44 GMT"} })
21772025-04-30T06:42:45.423ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/0/component/rot/cfpa
21782025-04-30T06:42:45.424ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "47ed2c52-ac79-42b2-9b8e-e3d1504783ef", "content-length": "728", "date": "Wed, 30 Apr 2025 06:42:44 GMT"} })
21792025-04-30T06:42:45.424ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/1
21802025-04-30T06:42:45.425ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "0402a554-8079-4fdc-8f3a-4b321abb9cac", "content-length": "734", "date": "Wed, 30 Apr 2025 06:42:44 GMT"} })
21812025-04-30T06:42:45.426ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/1/component/sp/caboose?firmware_slot=0
21822025-04-30T06:42:45.426ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
21832025-04-30T06:42:45.427ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {} }
21842025-04-30T06:42:45.428ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
21852025-04-30T06:42:45.433ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
21862025-04-30T06:42:45.433ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {} }
21872025-04-30T06:42:45.437ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
21882025-04-30T06:42:45.440ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "186b0f3c-ce90-455f-8a49-32f44bc2a46f", "content-length": "111", "date": "Wed, 30 Apr 2025 06:42:44 GMT"} })
21892025-04-30T06:42:45.440ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/1/component/sp/caboose?firmware_slot=1
21902025-04-30T06:42:45.441ZTRACtest_snapshot_basic (InternalApi): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = VpcSubnet { parent: Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ByName("oxide-services") }, key: 001de000-c470-4000-8000-000000000002, lookup_type: ByName("nexus") }
21912025-04-30T06:42:45.441ZTRACtest_snapshot_basic (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-4401-4000-8000-000000000000 resource_type = Project
21922025-04-30T06:42:45.441ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
21932025-04-30T06:42:45.441ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {} }
21942025-04-30T06:42:45.442ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
21952025-04-30T06:42:45.445ZTRACtest_snapshot_basic (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-5110-4000-8000-000000000001 resource_type = Silo
21962025-04-30T06:42:45.445ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
21972025-04-30T06:42:45.445ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {} }
21982025-04-30T06:42:45.446ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
21992025-04-30T06:42:45.446ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "7eaa36fd-21e4-4386-b907-631a45b067f5", "content-length": "111", "date": "Wed, 30 Apr 2025 06:42:44 GMT"} })
22002025-04-30T06:42:45.446ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/1/component/rot/caboose?firmware_slot=0
22012025-04-30T06:42:45.449ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "5c6d49f1-8874-418b-9d36-77435e237e09", "content-length": "170", "date": "Wed, 30 Apr 2025 06:42:44 GMT"} })
22022025-04-30T06:42:45.449ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/1/component/rot/caboose?firmware_slot=1
22032025-04-30T06:42:45.449ZTRACtest_snapshot_basic (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
22042025-04-30T06:42:45.449ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
22052025-04-30T06:42:45.449ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {} }
22062025-04-30T06:42:45.450ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
22072025-04-30T06:42:45.452ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "fbe9ce93-4f40-4621-bf45-3126d3ef942e", "content-length": "170", "date": "Wed, 30 Apr 2025 06:42:44 GMT"} })
22082025-04-30T06:42:45.452ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/1/component/stage0/caboose?firmware_slot=0
22092025-04-30T06:42:45.454ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
22102025-04-30T06:42:45.460ZDEBGtest_snapshot_basic (InternalApi): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = VpcSubnet { parent: Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ByName("oxide-services") }, key: 001de000-c470-4000-8000-000000000002, lookup_type: ByName("nexus") } result = Ok(())
22112025-04-30T06:42:45.460ZTRACtest_snapshot_basic (InternalApi): successful lookup of siloed resource "VpcSubnet" using built-in user
22122025-04-30T06:42:45.460ZTRACtest_snapshot_basic (InternalApi): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = VpcSubnet { parent: Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ByName("oxide-services") }, key: 001de000-c470-4000-8000-000000000002, lookup_type: ByName("nexus") }
22132025-04-30T06:42:45.460ZTRACtest_snapshot_basic (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-4401-4000-8000-000000000000 resource_type = Project
22142025-04-30T06:42:45.460ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
22152025-04-30T06:42:45.460ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {} }
22162025-04-30T06:42:45.461ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
22172025-04-30T06:42:45.463ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "102224fe-9131-4dec-bbfc-bf24207251c5", "content-length": "179", "date": "Wed, 30 Apr 2025 06:42:44 GMT"} })
22182025-04-30T06:42:45.463ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/1/component/stage0/caboose?firmware_slot=1
22192025-04-30T06:42:45.465ZTRACtest_snapshot_basic (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-5110-4000-8000-000000000001 resource_type = Silo
22202025-04-30T06:42:45.465ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
22212025-04-30T06:42:45.465ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {} }
22222025-04-30T06:42:45.465ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
22232025-04-30T06:42:45.466ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "68093897-e534-413a-84e2-05ffc244a518", "content-length": "179", "date": "Wed, 30 Apr 2025 06:42:44 GMT"} })
22242025-04-30T06:42:45.466ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/1/component/rot/cmpa
22252025-04-30T06:42:45.466ZTRACtest_snapshot_basic (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
22262025-04-30T06:42:45.466ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
22272025-04-30T06:42:45.466ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {} }
22282025-04-30T06:42:45.467ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
22292025-04-30T06:42:45.470ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
22302025-04-30T06:42:45.488ZDEBGtest_snapshot_basic (InternalApi): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = VpcSubnet { parent: Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ByName("oxide-services") }, key: 001de000-c470-4000-8000-000000000002, lookup_type: ByName("nexus") } result = Ok(())
22312025-04-30T06:42:45.488ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
22322025-04-30T06:42:45.488ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {} }
22332025-04-30T06:42:45.489ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
22342025-04-30T06:42:45.489ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "740394e8-4501-42d9-889b-b2bd384808e1", "content-length": "702", "date": "Wed, 30 Apr 2025 06:42:44 GMT"} })
22352025-04-30T06:42:45.489ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/1/component/rot/cfpa
22362025-04-30T06:42:45.491ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "a94cd13c-9eff-49ac-96c4-9386e3e86682", "content-length": "727", "date": "Wed, 30 Apr 2025 06:42:44 GMT"} })
22372025-04-30T06:42:45.491ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/1/component/rot/cfpa
22382025-04-30T06:42:45.492ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "4c27b6c4-073d-424f-947b-71d2d2fa440d", "content-length": "729", "date": "Wed, 30 Apr 2025 06:42:44 GMT"} })
22392025-04-30T06:42:45.492ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/1/component/rot/cfpa
22402025-04-30T06:42:45.493ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "db76269c-4741-4c01-8e1d-4ba4e9caeebf", "content-length": "728", "date": "Wed, 30 Apr 2025 06:42:44 GMT"} })
22412025-04-30T06:42:45.493ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/0
22422025-04-30T06:42:45.495ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
22432025-04-30T06:42:45.495ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {} }
22442025-04-30T06:42:45.495ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
22452025-04-30T06:42:45.498ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "ef72a3aa-f0a1-4211-9be1-b038ad1f0ed1", "content-length": "723", "date": "Wed, 30 Apr 2025 06:42:44 GMT"} })
22462025-04-30T06:42:45.499ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/0/component/sp/caboose?firmware_slot=0
22472025-04-30T06:42:45.499ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
22482025-04-30T06:42:45.499ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {} }
22492025-04-30T06:42:45.500ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
22502025-04-30T06:42:45.503ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "70cdacde-758a-4341-a212-8a9611f00428", "content-length": "109", "date": "Wed, 30 Apr 2025 06:42:44 GMT"} })
22512025-04-30T06:42:45.503ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/0/component/sp/caboose?firmware_slot=1
22522025-04-30T06:42:45.505ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
22532025-04-30T06:42:45.505ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {} }
22542025-04-30T06:42:45.506ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
22552025-04-30T06:42:45.508ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "a187b298-6fc1-401a-ae7c-e5ac07c73df5", "content-length": "109", "date": "Wed, 30 Apr 2025 06:42:44 GMT"} })
22562025-04-30T06:42:45.508ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/0/component/rot/caboose?firmware_slot=0
22572025-04-30T06:42:45.516ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
22582025-04-30T06:42:45.516ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {} }
22592025-04-30T06:42:45.517ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
22602025-04-30T06:42:45.526ZTRACtest_snapshot_basic (InternalApi): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = VpcSubnet { parent: Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ByName("oxide-services") }, key: 001de000-c470-4000-8000-000000000001, lookup_type: ByName("external-dns") }
22612025-04-30T06:42:45.526ZTRACtest_snapshot_basic (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-4401-4000-8000-000000000000 resource_type = Project
22622025-04-30T06:42:45.526ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
22632025-04-30T06:42:45.526ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {} }
22642025-04-30T06:42:45.527ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
22652025-04-30T06:42:45.530ZTRACtest_snapshot_basic (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-5110-4000-8000-000000000001 resource_type = Silo
22662025-04-30T06:42:45.530ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
22672025-04-30T06:42:45.531ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {} }
22682025-04-30T06:42:45.531ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
22692025-04-30T06:42:45.539ZTRACtest_snapshot_basic (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
22702025-04-30T06:42:45.539ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
22712025-04-30T06:42:45.539ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {} }
22722025-04-30T06:42:45.540ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
22732025-04-30T06:42:45.546ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
22742025-04-30T06:42:45.555ZDEBGtest_snapshot_basic (InternalApi): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = VpcSubnet { parent: Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ByName("oxide-services") }, key: 001de000-c470-4000-8000-000000000001, lookup_type: ByName("external-dns") } result = Ok(())
22752025-04-30T06:42:45.555ZTRACtest_snapshot_basic (InternalApi): successful lookup of siloed resource "VpcSubnet" using built-in user
22762025-04-30T06:42:45.555ZTRACtest_snapshot_basic (InternalApi): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = VpcSubnet { parent: Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ByName("oxide-services") }, key: 001de000-c470-4000-8000-000000000001, lookup_type: ByName("external-dns") }
22772025-04-30T06:42:45.555ZTRACtest_snapshot_basic (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-4401-4000-8000-000000000000 resource_type = Project
22782025-04-30T06:42:45.555ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
22792025-04-30T06:42:45.555ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {} }
22802025-04-30T06:42:45.557ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
22812025-04-30T06:42:45.563ZTRACtest_snapshot_basic (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-5110-4000-8000-000000000001 resource_type = Silo
22822025-04-30T06:42:45.564ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
22832025-04-30T06:42:45.564ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {} }
22842025-04-30T06:42:45.566ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
22852025-04-30T06:42:45.573ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "ba8792c3-1832-4151-99df-5a9a0128799d", "content-length": "169", "date": "Wed, 30 Apr 2025 06:42:44 GMT"} })
22862025-04-30T06:42:45.575ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/0/component/rot/caboose?firmware_slot=1
22872025-04-30T06:42:45.575ZTRACtest_snapshot_basic (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
22882025-04-30T06:42:45.576ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
22892025-04-30T06:42:45.576ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {} }
22902025-04-30T06:42:45.579ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
22912025-04-30T06:42:45.584ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
22922025-04-30T06:42:45.595ZDEBGtest_snapshot_basic (InternalApi): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = VpcSubnet { parent: Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ByName("oxide-services") }, key: 001de000-c470-4000-8000-000000000001, lookup_type: ByName("external-dns") } result = Ok(())
22932025-04-30T06:42:45.595ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
22942025-04-30T06:42:45.595ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {} }
22952025-04-30T06:42:45.596ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
22962025-04-30T06:42:45.598ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "65e631b1-bfed-4705-a53e-d1581009b7c6", "content-length": "169", "date": "Wed, 30 Apr 2025 06:42:44 GMT"} })
22972025-04-30T06:42:45.598ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/0/component/stage0/caboose?firmware_slot=0
22982025-04-30T06:42:45.604ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "5ad1d892-21ad-4044-95cc-64cec0849794", "content-length": "178", "date": "Wed, 30 Apr 2025 06:42:44 GMT"} })
22992025-04-30T06:42:45.604ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/0/component/stage0/caboose?firmware_slot=1
23002025-04-30T06:42:45.618ZDEBGtest_snapshot_basic (InternalApi): resolved names for firewall rules instance_interfaces = {} subnet_interfaces = {Name("external-dns"): [NetworkInterface { id: 94c5e059-ce01-475a-b1f0-c22bca5d9c16, kind: Service { id: 7e244b61-e4d7-4237-8aaa-92889da3453c }, name: Name("external-dns-7e244b61-e4d7-4237-8aaa-92889da3453c"), ip: 172.30.1.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 128, 0])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }], Name("nexus"): [NetworkInterface { id: cf6bcb96-a640-44ec-916e-463eed3d1741, kind: Service { id: 354a52d4-3078-4706-81ce-12c32f6bbeaa }, name: Name("nexus-354a52d4-3078-4706-81ce-12c32f6bbeaa"), ip: 172.30.2.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 128, 1])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }]} subnet_networks = {Name("external-dns"): [V4(Ipv4Network { addr: 172.30.1.0, prefix: 24 }), V6(Ipv6Network { addr: fd77:e9d2:9cd9:1::, prefix: 64 })], Name("nexus"): [V4(Ipv4Network { addr: 172.30.2.0, prefix: 24 }), V6(Ipv6Network { addr: fd77:e9d2:9cd9:2::, prefix: 64 })]} vpc_interfaces = {}
23012025-04-30T06:42:45.618ZTRACtest_snapshot_basic (InternalApi): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Fleet
23022025-04-30T06:42:45.618ZTRACtest_snapshot_basic (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
23032025-04-30T06:42:45.618ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
23042025-04-30T06:42:45.618ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {} }
23052025-04-30T06:42:45.619ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
23062025-04-30T06:42:45.622ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
23072025-04-30T06:42:45.626ZDEBGtest_snapshot_basic (InternalApi): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Fleet result = Ok(())
23082025-04-30T06:42:45.626ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
23092025-04-30T06:42:45.626ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {} }
23102025-04-30T06:42:45.626ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
23112025-04-30T06:42:45.630ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "ebc60b25-b8d5-4cd7-ab5f-e987522c8cf6", "content-length": "178", "date": "Wed, 30 Apr 2025 06:42:44 GMT"} })
23122025-04-30T06:42:45.630ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/0/component/rot/cmpa
23132025-04-30T06:42:45.630ZTRACtest_snapshot_basic (InternalApi): fetched allowlist from DB allowed = AllowList { id: 001de000-a110-4000-8000-000000000000, time_created: 2025-04-30T06:06:08.248500Z, time_modified: 2025-04-30T06:42:44.256494Z, allowed_ips: None }
23142025-04-30T06:42:45.630ZDEBGtest_snapshot_basic (InternalApi): Allowlist for user-facing services is set to allow any inbound traffic. Existing VPC firewall rules will not be modified.
23152025-04-30T06:42:45.631ZDEBGtest_snapshot_basic (InternalApi): resolved firewall rules for sled agents sled_agent_rules = [ResolvedVpcFirewallRule { action: Allow, direction: Inbound, filter_hosts: None, filter_ports: Some([L4PortRange("53")]), filter_protocols: Some([Udp]), priority: 65534, status: Enabled, targets: [NetworkInterface { id: 94c5e059-ce01-475a-b1f0-c22bca5d9c16, kind: Service { id: 7e244b61-e4d7-4237-8aaa-92889da3453c }, name: Name("external-dns-7e244b61-e4d7-4237-8aaa-92889da3453c"), ip: 172.30.1.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 128, 0])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }] }, ResolvedVpcFirewallRule { action: Allow, direction: Inbound, filter_hosts: None, filter_ports: Some([L4PortRange("80"), L4PortRange("443")]), filter_protocols: Some([Tcp]), priority: 65534, status: Enabled, targets: [NetworkInterface { id: cf6bcb96-a640-44ec-916e-463eed3d1741, kind: Service { id: 354a52d4-3078-4706-81ce-12c32f6bbeaa }, name: Name("nexus-354a52d4-3078-4706-81ce-12c32f6bbeaa"), ip: 172.30.2.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 128, 1])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }] }]
23162025-04-30T06:42:45.631ZDEBGtest_snapshot_basic (InternalApi): resolved 2 rules for sleds
23172025-04-30T06:42:45.633ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "3f3b1bc6-898f-453b-b638-853a1f75590b", "content-length": "702", "date": "Wed, 30 Apr 2025 06:42:44 GMT"} })
23182025-04-30T06:42:45.633ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/0/component/rot/cfpa
23192025-04-30T06:42:45.636ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "5da09089-2496-478e-89eb-3337b26193c8", "content-length": "727", "date": "Wed, 30 Apr 2025 06:42:44 GMT"} })
23202025-04-30T06:42:45.636ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/0/component/rot/cfpa
23212025-04-30T06:42:45.641ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "a54ff13f-af4c-443f-82b0-5b6483a05083", "content-length": "729", "date": "Wed, 30 Apr 2025 06:42:44 GMT"} })
23222025-04-30T06:42:45.648ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/0/component/rot/cfpa
23232025-04-30T06:42:45.650ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "a8c97f12-b8e7-43ee-83ec-4136a37d857b", "content-length": "728", "date": "Wed, 30 Apr 2025 06:42:44 GMT"} })
23242025-04-30T06:42:45.650ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/1
23252025-04-30T06:42:45.652ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "d54def28-cfce-4a12-b768-149415f40f68", "content-length": "723", "date": "Wed, 30 Apr 2025 06:42:44 GMT"} })
23262025-04-30T06:42:45.652ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/1/component/sp/caboose?firmware_slot=0
23272025-04-30T06:42:45.654ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "b2af7f27-f356-47be-997a-70fb754e6f10", "content-length": "109", "date": "Wed, 30 Apr 2025 06:42:44 GMT"} })
23282025-04-30T06:42:45.655ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/1/component/sp/caboose?firmware_slot=1
23292025-04-30T06:42:45.658ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "ac288aad-93c2-41bd-8366-760004b92c67", "content-length": "109", "date": "Wed, 30 Apr 2025 06:42:44 GMT"} })
23302025-04-30T06:42:45.658ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/1/component/rot/caboose?firmware_slot=0
23312025-04-30T06:42:45.662ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "3ff18601-7eff-44fa-a541-dafdbf742e3d", "content-length": "169", "date": "Wed, 30 Apr 2025 06:42:44 GMT"} })
23322025-04-30T06:42:45.662ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/1/component/rot/caboose?firmware_slot=1
23332025-04-30T06:42:45.664ZDEBGtest_snapshot_basic (InternalApi): resolved sleds for vpc oxide-services vpc_to_sled = [Sled { identity: SledIdentity { id: b6d65341-167c-41df-9b5c-41cded99c229, time_created: 2025-04-30T06:42:38.543918Z, time_modified: 2025-04-30T06:42:38.543918Z }, time_deleted: None, rcgen: Generation(Generation(1)), rack_id: c19a698f-c6f9-4a17-ae30-20d711b8f7dc, is_scrimlet: true, serial_number: "sim-b6d65341", part_number: "Unknown", revision: SqlU32(0), usable_hardware_threads: SqlU32(16), usable_physical_ram: ByteCount(ByteCount(34359738368)), reservoir_size: ByteCount(ByteCount(17179869184)), ip: ::1, port: SqlU16(41923), last_used_address: ::ffff, policy: InService, state: Active, sled_agent_gen: Generation(Generation(1)), repo_depot_port: SqlU16(46799) }]
23342025-04-30T06:42:45.664ZDEBGtest_snapshot_basic (InternalApi): sending firewall rules to sled agents
23352025-04-30T06:42:45.664ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
23362025-04-30T06:42:45.664ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {} }
23372025-04-30T06:42:45.666ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
23382025-04-30T06:42:45.669ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "66f10ffa-520e-419f-8347-555ddee211df", "content-length": "169", "date": "Wed, 30 Apr 2025 06:42:44 GMT"} })
23392025-04-30T06:42:45.669ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/1/component/stage0/caboose?firmware_slot=0
23402025-04-30T06:42:45.669ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Sled { parent: Fleet, key: b6d65341-167c-41df-9b5c-41cded99c229, lookup_type: ById(b6d65341-167c-41df-9b5c-41cded99c229) }
23412025-04-30T06:42:45.669ZTRACtest_snapshot_basic (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
23422025-04-30T06:42:45.669ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
23432025-04-30T06:42:45.669ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {} }
23442025-04-30T06:42:45.671ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
23452025-04-30T06:42:45.673ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
23462025-04-30T06:42:45.680ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Sled { parent: Fleet, key: b6d65341-167c-41df-9b5c-41cded99c229, lookup_type: ById(b6d65341-167c-41df-9b5c-41cded99c229) } result = Ok(())
23472025-04-30T06:42:45.756ZDEBGtest_snapshot_basic (InternalApi): client request SledAgent = b6d65341-167c-41df-9b5c-41cded99c229 body = Some(Body) method = PUT uri = http://[::1]:41923/vpc/001de000-074c-4000-8000-000000000000/firewall/rules
23482025-04-30T06:42:45.757ZINFOtest_snapshot_basic (dropshot): accepted connection local_addr = [::1]:41923 remote_addr = [::1]:50403 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
23492025-04-30T06:42:45.759ZTRACtest_snapshot_basic (dropshot): incoming request local_addr = [::1]:41923 method = PUT remote_addr = [::1]:50403 req_id = 88f39e37-666c-4234-926f-7f30feafc93c sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /vpc/001de000-074c-4000-8000-000000000000/firewall/rules
23502025-04-30T06:42:45.760ZINFOtest_snapshot_basic (dropshot): request completed latency_us = 898 local_addr = [::1]:41923 method = PUT remote_addr = [::1]:50403 req_id = 88f39e37-666c-4234-926f-7f30feafc93c response_code = 204 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /vpc/001de000-074c-4000-8000-000000000000/firewall/rules
23512025-04-30T06:42:45.761ZDEBGtest_snapshot_basic (InternalApi): client response SledAgent = b6d65341-167c-41df-9b5c-41cded99c229 result = Ok(Response { url: "http://[::1]:41923/vpc/001de000-074c-4000-8000-000000000000/firewall/rules", status: 204, headers: {"x-request-id": "88f39e37-666c-4234-926f-7f30feafc93c", "date": "Wed, 30 Apr 2025 06:42:44 GMT"} })
23522025-04-30T06:42:45.764ZINFOtest_snapshot_basic (InternalApi): updated firewall rules on 1 sleds
23532025-04-30T06:42:45.764ZINFOtest_snapshot_basic (nexus): plumbed initial IP allowlist
23542025-04-30T06:42:45.821ZDEBGtest_snapshot_basic (dropshot_external): registered endpoint local_addr = 127.0.0.1:44640 method = GET path = / versions = all
23552025-04-30T06:42:45.821ZDEBGtest_snapshot_basic (dropshot_external): registered endpoint local_addr = 127.0.0.1:44640 method = GET path = /access versions = all
23562025-04-30T06:42:45.821ZDEBGtest_snapshot_basic (dropshot_external): registered endpoint local_addr = 127.0.0.1:44640 method = GET path = /assets/{path} versions = all
23572025-04-30T06:42:45.821ZDEBGtest_snapshot_basic (dropshot_external): registered endpoint local_addr = 127.0.0.1:44640 method = POST path = /device/auth versions = all
23582025-04-30T06:42:45.821ZDEBGtest_snapshot_basic (dropshot_external): registered endpoint local_addr = 127.0.0.1:44640 method = POST path = /device/confirm versions = all
23592025-04-30T06:42:45.821ZDEBGtest_snapshot_basic (dropshot_external): registered endpoint local_addr = 127.0.0.1:44640 method = GET path = /device/success versions = all
23602025-04-30T06:42:45.821ZDEBGtest_snapshot_basic (dropshot_external): registered endpoint local_addr = 127.0.0.1:44640 method = POST path = /device/token versions = all
23612025-04-30T06:42:45.821ZDEBGtest_snapshot_basic (dropshot_external): registered endpoint local_addr = 127.0.0.1:44640 method = GET path = /device/verify versions = all
23622025-04-30T06:42:45.821ZDEBGtest_snapshot_basic (dropshot_external): registered endpoint local_addr = 127.0.0.1:44640 method = GET path = /experimental/v1/probes versions = all
23632025-04-30T06:42:45.821ZDEBGtest_snapshot_basic (dropshot_external): registered endpoint local_addr = 127.0.0.1:44640 method = POST path = /experimental/v1/probes versions = all
23642025-04-30T06:42:45.821ZDEBGtest_snapshot_basic (dropshot_external): registered endpoint local_addr = 127.0.0.1:44640 method = DELETE path = /experimental/v1/probes/{probe} versions = all
23652025-04-30T06:42:45.821ZDEBGtest_snapshot_basic (dropshot_external): registered endpoint local_addr = 127.0.0.1:44640 method = GET path = /experimental/v1/probes/{probe} versions = all
23662025-04-30T06:42:45.821ZDEBGtest_snapshot_basic (dropshot_external): registered endpoint local_addr = 127.0.0.1:44640 method = GET path = /experimental/v1/system/support-bundles versions = all
23672025-04-30T06:42:45.821ZDEBGtest_snapshot_basic (dropshot_external): registered endpoint local_addr = 127.0.0.1:44640 method = POST path = /experimental/v1/system/support-bundles versions = all
23682025-04-30T06:42:45.821ZDEBGtest_snapshot_basic (dropshot_external): registered endpoint local_addr = 127.0.0.1:44640 method = DELETE path = /experimental/v1/system/support-bundles/{support_bundle} versions = all
23692025-04-30T06:42:45.821ZDEBGtest_snapshot_basic (dropshot_external): registered endpoint local_addr = 127.0.0.1:44640 method = GET path = /experimental/v1/system/support-bundles/{support_bundle} versions = all
23702025-04-30T06:42:45.821ZDEBGtest_snapshot_basic (dropshot_external): registered endpoint local_addr = 127.0.0.1:44640 method = GET path = /experimental/v1/system/support-bundles/{support_bundle}/download versions = all
23712025-04-30T06:42:45.821ZDEBGtest_snapshot_basic (dropshot_external): registered endpoint local_addr = 127.0.0.1:44640 method = HEAD path = /experimental/v1/system/support-bundles/{support_bundle}/download versions = all
23722025-04-30T06:42:45.821ZDEBGtest_snapshot_basic (dropshot_external): registered endpoint local_addr = 127.0.0.1:44640 method = GET path = /experimental/v1/system/support-bundles/{support_bundle}/download/{file} versions = all
23732025-04-30T06:42:45.821ZDEBGtest_snapshot_basic (dropshot_external): registered endpoint local_addr = 127.0.0.1:44640 method = HEAD path = /experimental/v1/system/support-bundles/{support_bundle}/download/{file} versions = all
23742025-04-30T06:42:45.821ZDEBGtest_snapshot_basic (dropshot_external): registered endpoint local_addr = 127.0.0.1:44640 method = GET path = /experimental/v1/system/support-bundles/{support_bundle}/index versions = all
23752025-04-30T06:42:45.821ZDEBGtest_snapshot_basic (dropshot_external): registered endpoint local_addr = 127.0.0.1:44640 method = GET path = /images versions = all
23762025-04-30T06:42:45.821ZDEBGtest_snapshot_basic (dropshot_external): registered endpoint local_addr = 127.0.0.1:44640 method = GET path = /login versions = all
23772025-04-30T06:42:45.822ZDEBGtest_snapshot_basic (dropshot_external): registered endpoint local_addr = 127.0.0.1:44640 method = GET path = /login/{silo_name}/local versions = all
23782025-04-30T06:42:45.822ZDEBGtest_snapshot_basic (dropshot_external): registered endpoint local_addr = 127.0.0.1:44640 method = GET path = /login/{silo_name}/saml/{provider_name} versions = all
23792025-04-30T06:42:45.822ZDEBGtest_snapshot_basic (dropshot_external): registered endpoint local_addr = 127.0.0.1:44640 method = POST path = /login/{silo_name}/saml/{provider_name} versions = all
23802025-04-30T06:42:45.822ZDEBGtest_snapshot_basic (dropshot_external): registered endpoint local_addr = 127.0.0.1:44640 method = GET path = /login/{silo_name}/saml/{provider_name}/redirect versions = all
23812025-04-30T06:42:45.822ZDEBGtest_snapshot_basic (dropshot_external): registered endpoint local_addr = 127.0.0.1:44640 method = GET path = /lookup/{path} versions = all
23822025-04-30T06:42:45.822ZDEBGtest_snapshot_basic (dropshot_external): registered endpoint local_addr = 127.0.0.1:44640 method = GET path = /projects/{path} versions = all
23832025-04-30T06:42:45.822ZDEBGtest_snapshot_basic (dropshot_external): registered endpoint local_addr = 127.0.0.1:44640 method = GET path = /projects-new versions = all
23842025-04-30T06:42:45.822ZDEBGtest_snapshot_basic (dropshot_external): registered endpoint local_addr = 127.0.0.1:44640 method = GET path = /settings/{path} versions = all
23852025-04-30T06:42:45.822ZDEBGtest_snapshot_basic (dropshot_external): registered endpoint local_addr = 127.0.0.1:44640 method = GET path = /system/{path} versions = all
23862025-04-30T06:42:45.822ZDEBGtest_snapshot_basic (dropshot_external): registered endpoint local_addr = 127.0.0.1:44640 method = GET path = /utilization versions = all
23872025-04-30T06:42:45.822ZDEBGtest_snapshot_basic (dropshot_external): registered endpoint local_addr = 127.0.0.1:44640 method = GET path = /v1/affinity-groups versions = all
23882025-04-30T06:42:45.822ZDEBGtest_snapshot_basic (dropshot_external): registered endpoint local_addr = 127.0.0.1:44640 method = POST path = /v1/affinity-groups versions = all
23892025-04-30T06:42:45.822ZDEBGtest_snapshot_basic (dropshot_external): registered endpoint local_addr = 127.0.0.1:44640 method = DELETE path = /v1/affinity-groups/{affinity_group} versions = all
23902025-04-30T06:42:45.822ZDEBGtest_snapshot_basic (dropshot_external): registered endpoint local_addr = 127.0.0.1:44640 method = GET path = /v1/affinity-groups/{affinity_group} versions = all
23912025-04-30T06:42:45.822ZDEBGtest_snapshot_basic (dropshot_external): registered endpoint local_addr = 127.0.0.1:44640 method = PUT path = /v1/affinity-groups/{affinity_group} versions = all
23922025-04-30T06:42:45.822ZDEBGtest_snapshot_basic (dropshot_external): registered endpoint local_addr = 127.0.0.1:44640 method = GET path = /v1/affinity-groups/{affinity_group}/members versions = all
23932025-04-30T06:42:45.822ZDEBGtest_snapshot_basic (dropshot_external): registered endpoint local_addr = 127.0.0.1:44640 method = DELETE path = /v1/affinity-groups/{affinity_group}/members/instance/{instance} versions = all
23942025-04-30T06:42:45.822ZDEBGtest_snapshot_basic (dropshot_external): registered endpoint local_addr = 127.0.0.1:44640 method = GET path = /v1/affinity-groups/{affinity_group}/members/instance/{instance} versions = all
23952025-04-30T06:42:45.822ZDEBGtest_snapshot_basic (dropshot_external): registered endpoint local_addr = 127.0.0.1:44640 method = POST path = /v1/affinity-groups/{affinity_group}/members/instance/{instance} versions = all
23962025-04-30T06:42:45.822ZDEBGtest_snapshot_basic (dropshot_external): registered endpoint local_addr = 127.0.0.1:44640 method = GET path = /v1/anti-affinity-groups versions = all
23972025-04-30T06:42:45.822ZDEBGtest_snapshot_basic (dropshot_external): registered endpoint local_addr = 127.0.0.1:44640 method = POST path = /v1/anti-affinity-groups versions = all
23982025-04-30T06:42:45.822ZDEBGtest_snapshot_basic (dropshot_external): registered endpoint local_addr = 127.0.0.1:44640 method = DELETE path = /v1/anti-affinity-groups/{anti_affinity_group} versions = all
23992025-04-30T06:42:45.822ZDEBGtest_snapshot_basic (dropshot_external): registered endpoint local_addr = 127.0.0.1:44640 method = GET path = /v1/anti-affinity-groups/{anti_affinity_group} versions = all
24002025-04-30T06:42:45.822ZDEBGtest_snapshot_basic (dropshot_external): registered endpoint local_addr = 127.0.0.1:44640 method = PUT path = /v1/anti-affinity-groups/{anti_affinity_group} versions = all
24012025-04-30T06:42:45.822ZDEBGtest_snapshot_basic (dropshot_external): registered endpoint local_addr = 127.0.0.1:44640 method = GET path = /v1/anti-affinity-groups/{anti_affinity_group}/members versions = all
24022025-04-30T06:42:45.822ZDEBGtest_snapshot_basic (dropshot_external): registered endpoint local_addr = 127.0.0.1:44640 method = DELETE path = /v1/anti-affinity-groups/{anti_affinity_group}/members/instance/{instance} versions = all
24032025-04-30T06:42:45.822ZDEBGtest_snapshot_basic (dropshot_external): registered endpoint local_addr = 127.0.0.1:44640 method = GET path = /v1/anti-affinity-groups/{anti_affinity_group}/members/instance/{instance} versions = all
24042025-04-30T06:42:45.822ZDEBGtest_snapshot_basic (dropshot_external): registered endpoint local_addr = 127.0.0.1:44640 method = POST path = /v1/anti-affinity-groups/{anti_affinity_group}/members/instance/{instance} versions = all
24052025-04-30T06:42:45.822ZDEBGtest_snapshot_basic (dropshot_external): registered endpoint local_addr = 127.0.0.1:44640 method = GET path = /v1/certificates versions = all
24062025-04-30T06:42:45.822ZDEBGtest_snapshot_basic (dropshot_external): registered endpoint local_addr = 127.0.0.1:44640 method = POST path = /v1/certificates versions = all
24072025-04-30T06:42:45.823ZDEBGtest_snapshot_basic (dropshot_external): registered endpoint local_addr = 127.0.0.1:44640 method = DELETE path = /v1/certificates/{certificate} versions = all
24082025-04-30T06:42:45.823ZDEBGtest_snapshot_basic (dropshot_external): registered endpoint local_addr = 127.0.0.1:44640 method = GET path = /v1/certificates/{certificate} versions = all
24092025-04-30T06:42:45.823ZDEBGtest_snapshot_basic (dropshot_external): registered endpoint local_addr = 127.0.0.1:44640 method = GET path = /v1/disks versions = all
24102025-04-30T06:42:45.823ZDEBGtest_snapshot_basic (dropshot_external): registered endpoint local_addr = 127.0.0.1:44640 method = POST path = /v1/disks versions = all
24112025-04-30T06:42:45.823ZDEBGtest_snapshot_basic (dropshot_external): registered endpoint local_addr = 127.0.0.1:44640 method = DELETE path = /v1/disks/{disk} versions = all
24122025-04-30T06:42:45.823ZDEBGtest_snapshot_basic (dropshot_external): registered endpoint local_addr = 127.0.0.1:44640 method = GET path = /v1/disks/{disk} versions = all
24132025-04-30T06:42:45.823ZDEBGtest_snapshot_basic (dropshot_external): registered endpoint local_addr = 127.0.0.1:44640 method = POST path = /v1/disks/{disk}/bulk-write versions = all
24142025-04-30T06:42:45.823ZDEBGtest_snapshot_basic (dropshot_external): registered endpoint local_addr = 127.0.0.1:44640 method = POST path = /v1/disks/{disk}/bulk-write-start versions = all
24152025-04-30T06:42:45.823ZDEBGtest_snapshot_basic (dropshot_external): registered endpoint local_addr = 127.0.0.1:44640 method = POST path = /v1/disks/{disk}/bulk-write-stop versions = all
24162025-04-30T06:42:45.823ZDEBGtest_snapshot_basic (dropshot_external): registered endpoint local_addr = 127.0.0.1:44640 method = POST path = /v1/disks/{disk}/finalize versions = all
24172025-04-30T06:42:45.823ZDEBGtest_snapshot_basic (dropshot_external): registered endpoint local_addr = 127.0.0.1:44640 method = GET path = /v1/disks/{disk}/metrics/{metric} versions = all
24182025-04-30T06:42:45.823ZDEBGtest_snapshot_basic (dropshot_external): registered endpoint local_addr = 127.0.0.1:44640 method = GET path = /v1/floating-ips versions = all
24192025-04-30T06:42:45.823ZDEBGtest_snapshot_basic (dropshot_external): registered endpoint local_addr = 127.0.0.1:44640 method = POST path = /v1/floating-ips versions = all
24202025-04-30T06:42:45.823ZDEBGtest_snapshot_basic (dropshot_external): registered endpoint local_addr = 127.0.0.1:44640 method = DELETE path = /v1/floating-ips/{floating_ip} versions = all
24212025-04-30T06:42:45.823ZDEBGtest_snapshot_basic (dropshot_external): registered endpoint local_addr = 127.0.0.1:44640 method = GET path = /v1/floating-ips/{floating_ip} versions = all
24222025-04-30T06:42:45.823ZDEBGtest_snapshot_basic (dropshot_external): registered endpoint local_addr = 127.0.0.1:44640 method = PUT path = /v1/floating-ips/{floating_ip} versions = all
24232025-04-30T06:42:45.823ZDEBGtest_snapshot_basic (dropshot_external): registered endpoint local_addr = 127.0.0.1:44640 method = POST path = /v1/floating-ips/{floating_ip}/attach versions = all
24242025-04-30T06:42:45.823ZDEBGtest_snapshot_basic (dropshot_external): registered endpoint local_addr = 127.0.0.1:44640 method = POST path = /v1/floating-ips/{floating_ip}/detach versions = all
24252025-04-30T06:42:45.823ZDEBGtest_snapshot_basic (dropshot_external): registered endpoint local_addr = 127.0.0.1:44640 method = GET path = /v1/groups versions = all
24262025-04-30T06:42:45.823ZDEBGtest_snapshot_basic (dropshot_external): registered endpoint local_addr = 127.0.0.1:44640 method = GET path = /v1/groups/{group_id} versions = all
24272025-04-30T06:42:45.823ZDEBGtest_snapshot_basic (dropshot_external): registered endpoint local_addr = 127.0.0.1:44640 method = GET path = /v1/images versions = all
24282025-04-30T06:42:45.823ZDEBGtest_snapshot_basic (dropshot_external): registered endpoint local_addr = 127.0.0.1:44640 method = POST path = /v1/images versions = all
24292025-04-30T06:42:45.823ZDEBGtest_snapshot_basic (dropshot_external): registered endpoint local_addr = 127.0.0.1:44640 method = DELETE path = /v1/images/{image} versions = all
24302025-04-30T06:42:45.823ZDEBGtest_snapshot_basic (dropshot_external): registered endpoint local_addr = 127.0.0.1:44640 method = GET path = /v1/images/{image} versions = all
24312025-04-30T06:42:45.823ZDEBGtest_snapshot_basic (dropshot_external): registered endpoint local_addr = 127.0.0.1:44640 method = POST path = /v1/images/{image}/demote versions = all
24322025-04-30T06:42:45.823ZDEBGtest_snapshot_basic (dropshot_external): registered endpoint local_addr = 127.0.0.1:44640 method = POST path = /v1/images/{image}/promote versions = all
24332025-04-30T06:42:45.823ZDEBGtest_snapshot_basic (dropshot_external): registered endpoint local_addr = 127.0.0.1:44640 method = GET path = /v1/instances versions = all
24342025-04-30T06:42:45.823ZDEBGtest_snapshot_basic (dropshot_external): registered endpoint local_addr = 127.0.0.1:44640 method = POST path = /v1/instances versions = all
24352025-04-30T06:42:45.823ZDEBGtest_snapshot_basic (dropshot_external): registered endpoint local_addr = 127.0.0.1:44640 method = DELETE path = /v1/instances/{instance} versions = all
24362025-04-30T06:42:45.823ZDEBGtest_snapshot_basic (dropshot_external): registered endpoint local_addr = 127.0.0.1:44640 method = GET path = /v1/instances/{instance} versions = all
24372025-04-30T06:42:45.823ZDEBGtest_snapshot_basic (dropshot_external): registered endpoint local_addr = 127.0.0.1:44640 method = PUT path = /v1/instances/{instance} versions = all
24382025-04-30T06:42:45.823ZDEBGtest_snapshot_basic (dropshot_external): registered endpoint local_addr = 127.0.0.1:44640 method = GET path = /v1/instances/{instance}/affinity-groups versions = all
24392025-04-30T06:42:45.823ZDEBGtest_snapshot_basic (dropshot_external): registered endpoint local_addr = 127.0.0.1:44640 method = GET path = /v1/instances/{instance}/anti-affinity-groups versions = all
24402025-04-30T06:42:45.823ZDEBGtest_snapshot_basic (dropshot_external): registered endpoint local_addr = 127.0.0.1:44640 method = GET path = /v1/instances/{instance}/disks versions = all
24412025-04-30T06:42:45.823ZDEBGtest_snapshot_basic (dropshot_external): registered endpoint local_addr = 127.0.0.1:44640 method = POST path = /v1/instances/{instance}/disks/attach versions = all
24422025-04-30T06:42:45.823ZDEBGtest_snapshot_basic (dropshot_external): registered endpoint local_addr = 127.0.0.1:44640 method = POST path = /v1/instances/{instance}/disks/detach versions = all
24432025-04-30T06:42:45.823ZDEBGtest_snapshot_basic (dropshot_external): registered endpoint local_addr = 127.0.0.1:44640 method = GET path = /v1/instances/{instance}/external-ips versions = all
24442025-04-30T06:42:45.823ZDEBGtest_snapshot_basic (dropshot_external): registered endpoint local_addr = 127.0.0.1:44640 method = DELETE path = /v1/instances/{instance}/external-ips/ephemeral versions = all
24452025-04-30T06:42:45.823ZDEBGtest_snapshot_basic (dropshot_external): registered endpoint local_addr = 127.0.0.1:44640 method = POST path = /v1/instances/{instance}/external-ips/ephemeral versions = all
24462025-04-30T06:42:45.823ZDEBGtest_snapshot_basic (dropshot_external): registered endpoint local_addr = 127.0.0.1:44640 method = POST path = /v1/instances/{instance}/reboot versions = all
24472025-04-30T06:42:45.824ZDEBGtest_snapshot_basic (dropshot_external): registered endpoint local_addr = 127.0.0.1:44640 method = GET path = /v1/instances/{instance}/serial-console versions = all
24482025-04-30T06:42:45.824ZDEBGtest_snapshot_basic (dropshot_external): registered endpoint local_addr = 127.0.0.1:44640 method = GET path = /v1/instances/{instance}/serial-console/stream versions = all
24492025-04-30T06:42:45.824ZDEBGtest_snapshot_basic (dropshot_external): registered endpoint local_addr = 127.0.0.1:44640 method = GET path = /v1/instances/{instance}/ssh-public-keys versions = all
24502025-04-30T06:42:45.824ZDEBGtest_snapshot_basic (dropshot_external): registered endpoint local_addr = 127.0.0.1:44640 method = POST path = /v1/instances/{instance}/start versions = all
24512025-04-30T06:42:45.824ZDEBGtest_snapshot_basic (dropshot_external): registered endpoint local_addr = 127.0.0.1:44640 method = POST path = /v1/instances/{instance}/stop versions = all
24522025-04-30T06:42:45.824ZDEBGtest_snapshot_basic (dropshot_external): registered endpoint local_addr = 127.0.0.1:44640 method = GET path = /v1/internet-gateway-ip-addresses versions = all
24532025-04-30T06:42:45.824ZDEBGtest_snapshot_basic (dropshot_external): registered endpoint local_addr = 127.0.0.1:44640 method = POST path = /v1/internet-gateway-ip-addresses versions = all
24542025-04-30T06:42:45.824ZDEBGtest_snapshot_basic (dropshot_external): registered endpoint local_addr = 127.0.0.1:44640 method = DELETE path = /v1/internet-gateway-ip-addresses/{address} versions = all
24552025-04-30T06:42:45.824ZDEBGtest_snapshot_basic (dropshot_external): registered endpoint local_addr = 127.0.0.1:44640 method = GET path = /v1/internet-gateway-ip-pools versions = all
24562025-04-30T06:42:45.824ZDEBGtest_snapshot_basic (dropshot_external): registered endpoint local_addr = 127.0.0.1:44640 method = POST path = /v1/internet-gateway-ip-pools versions = all
24572025-04-30T06:42:45.824ZDEBGtest_snapshot_basic (dropshot_external): registered endpoint local_addr = 127.0.0.1:44640 method = DELETE path = /v1/internet-gateway-ip-pools/{pool} versions = all
24582025-04-30T06:42:45.824ZDEBGtest_snapshot_basic (dropshot_external): registered endpoint local_addr = 127.0.0.1:44640 method = GET path = /v1/internet-gateways versions = all
24592025-04-30T06:42:45.824ZDEBGtest_snapshot_basic (dropshot_external): registered endpoint local_addr = 127.0.0.1:44640 method = POST path = /v1/internet-gateways versions = all
24602025-04-30T06:42:45.824ZDEBGtest_snapshot_basic (dropshot_external): registered endpoint local_addr = 127.0.0.1:44640 method = DELETE path = /v1/internet-gateways/{gateway} versions = all
24612025-04-30T06:42:45.824ZDEBGtest_snapshot_basic (dropshot_external): registered endpoint local_addr = 127.0.0.1:44640 method = GET path = /v1/internet-gateways/{gateway} versions = all
24622025-04-30T06:42:45.824ZDEBGtest_snapshot_basic (dropshot_external): registered endpoint local_addr = 127.0.0.1:44640 method = GET path = /v1/ip-pools versions = all
24632025-04-30T06:42:45.824ZDEBGtest_snapshot_basic (dropshot_external): registered endpoint local_addr = 127.0.0.1:44640 method = GET path = /v1/ip-pools/{pool} versions = all
24642025-04-30T06:42:45.824ZDEBGtest_snapshot_basic (dropshot_external): registered endpoint local_addr = 127.0.0.1:44640 method = POST path = /v1/login/{silo_name}/local versions = all
24652025-04-30T06:42:45.824ZDEBGtest_snapshot_basic (dropshot_external): registered endpoint local_addr = 127.0.0.1:44640 method = POST path = /v1/logout versions = all
24662025-04-30T06:42:45.824ZDEBGtest_snapshot_basic (dropshot_external): registered endpoint local_addr = 127.0.0.1:44640 method = GET path = /v1/me versions = all
24672025-04-30T06:42:45.824ZDEBGtest_snapshot_basic (dropshot_external): registered endpoint local_addr = 127.0.0.1:44640 method = GET path = /v1/me/groups versions = all
24682025-04-30T06:42:45.824ZDEBGtest_snapshot_basic (dropshot_external): registered endpoint local_addr = 127.0.0.1:44640 method = GET path = /v1/me/ssh-keys versions = all
24692025-04-30T06:42:45.824ZDEBGtest_snapshot_basic (dropshot_external): registered endpoint local_addr = 127.0.0.1:44640 method = POST path = /v1/me/ssh-keys versions = all
24702025-04-30T06:42:45.824ZDEBGtest_snapshot_basic (dropshot_external): registered endpoint local_addr = 127.0.0.1:44640 method = DELETE path = /v1/me/ssh-keys/{ssh_key} versions = all
24712025-04-30T06:42:45.824ZDEBGtest_snapshot_basic (dropshot_external): registered endpoint local_addr = 127.0.0.1:44640 method = GET path = /v1/me/ssh-keys/{ssh_key} versions = all
24722025-04-30T06:42:45.824ZDEBGtest_snapshot_basic (dropshot_external): registered endpoint local_addr = 127.0.0.1:44640 method = GET path = /v1/metrics/{metric_name} versions = all
24732025-04-30T06:42:45.824ZDEBGtest_snapshot_basic (dropshot_external): registered endpoint local_addr = 127.0.0.1:44640 method = GET path = /v1/network-interfaces versions = all
24742025-04-30T06:42:45.824ZDEBGtest_snapshot_basic (dropshot_external): registered endpoint local_addr = 127.0.0.1:44640 method = POST path = /v1/network-interfaces versions = all
24752025-04-30T06:42:45.824ZDEBGtest_snapshot_basic (dropshot_external): registered endpoint local_addr = 127.0.0.1:44640 method = DELETE path = /v1/network-interfaces/{interface} versions = all
24762025-04-30T06:42:45.824ZDEBGtest_snapshot_basic (dropshot_external): registered endpoint local_addr = 127.0.0.1:44640 method = GET path = /v1/network-interfaces/{interface} versions = all
24772025-04-30T06:42:45.824ZDEBGtest_snapshot_basic (dropshot_external): registered endpoint local_addr = 127.0.0.1:44640 method = PUT path = /v1/network-interfaces/{interface} versions = all
24782025-04-30T06:42:45.824ZDEBGtest_snapshot_basic (dropshot_external): registered endpoint local_addr = 127.0.0.1:44640 method = GET path = /v1/ping versions = all
24792025-04-30T06:42:45.824ZDEBGtest_snapshot_basic (dropshot_external): registered endpoint local_addr = 127.0.0.1:44640 method = GET path = /v1/policy versions = all
24802025-04-30T06:42:45.824ZDEBGtest_snapshot_basic (dropshot_external): registered endpoint local_addr = 127.0.0.1:44640 method = PUT path = /v1/policy versions = all
24812025-04-30T06:42:45.824ZDEBGtest_snapshot_basic (dropshot_external): registered endpoint local_addr = 127.0.0.1:44640 method = GET path = /v1/projects versions = all
24822025-04-30T06:42:45.824ZDEBGtest_snapshot_basic (dropshot_external): registered endpoint local_addr = 127.0.0.1:44640 method = POST path = /v1/projects versions = all
24832025-04-30T06:42:45.825ZDEBGtest_snapshot_basic (dropshot_external): registered endpoint local_addr = 127.0.0.1:44640 method = DELETE path = /v1/projects/{project} versions = all
24842025-04-30T06:42:45.825ZDEBGtest_snapshot_basic (dropshot_external): registered endpoint local_addr = 127.0.0.1:44640 method = GET path = /v1/projects/{project} versions = all
24852025-04-30T06:42:45.825ZDEBGtest_snapshot_basic (dropshot_external): registered endpoint local_addr = 127.0.0.1:44640 method = PUT path = /v1/projects/{project} versions = all
24862025-04-30T06:42:45.825ZDEBGtest_snapshot_basic (dropshot_external): registered endpoint local_addr = 127.0.0.1:44640 method = GET path = /v1/projects/{project}/policy versions = all
24872025-04-30T06:42:45.825ZDEBGtest_snapshot_basic (dropshot_external): registered endpoint local_addr = 127.0.0.1:44640 method = PUT path = /v1/projects/{project}/policy versions = all
24882025-04-30T06:42:45.825ZDEBGtest_snapshot_basic (dropshot_external): registered endpoint local_addr = 127.0.0.1:44640 method = GET path = /v1/snapshots versions = all
24892025-04-30T06:42:45.825ZDEBGtest_snapshot_basic (dropshot_external): registered endpoint local_addr = 127.0.0.1:44640 method = POST path = /v1/snapshots versions = all
24902025-04-30T06:42:45.825ZDEBGtest_snapshot_basic (dropshot_external): registered endpoint local_addr = 127.0.0.1:44640 method = DELETE path = /v1/snapshots/{snapshot} versions = all
24912025-04-30T06:42:45.825ZDEBGtest_snapshot_basic (dropshot_external): registered endpoint local_addr = 127.0.0.1:44640 method = GET path = /v1/snapshots/{snapshot} versions = all
24922025-04-30T06:42:45.825ZDEBGtest_snapshot_basic (dropshot_external): registered endpoint local_addr = 127.0.0.1:44640 method = GET path = /v1/system/hardware/disks versions = all
24932025-04-30T06:42:45.825ZDEBGtest_snapshot_basic (dropshot_external): registered endpoint local_addr = 127.0.0.1:44640 method = GET path = /v1/system/hardware/disks/{disk_id} versions = all
24942025-04-30T06:42:45.825ZDEBGtest_snapshot_basic (dropshot_external): registered endpoint local_addr = 127.0.0.1:44640 method = GET path = /v1/system/hardware/rack-switch-port/{rack_id}/{switch_location}/{port}/lldp/neighbors versions = all
24952025-04-30T06:42:45.825ZDEBGtest_snapshot_basic (dropshot_external): registered endpoint local_addr = 127.0.0.1:44640 method = GET path = /v1/system/hardware/racks versions = all
24962025-04-30T06:42:45.825ZDEBGtest_snapshot_basic (dropshot_external): registered endpoint local_addr = 127.0.0.1:44640 method = GET path = /v1/system/hardware/racks/{rack_id} versions = all
24972025-04-30T06:42:45.825ZDEBGtest_snapshot_basic (dropshot_external): registered endpoint local_addr = 127.0.0.1:44640 method = GET path = /v1/system/hardware/sleds versions = all
24982025-04-30T06:42:45.825ZDEBGtest_snapshot_basic (dropshot_external): registered endpoint local_addr = 127.0.0.1:44640 method = POST path = /v1/system/hardware/sleds versions = all
24992025-04-30T06:42:45.825ZDEBGtest_snapshot_basic (dropshot_external): registered endpoint local_addr = 127.0.0.1:44640 method = GET path = /v1/system/hardware/sleds/{sled_id} versions = all
25002025-04-30T06:42:45.825ZDEBGtest_snapshot_basic (dropshot_external): registered endpoint local_addr = 127.0.0.1:44640 method = GET path = /v1/system/hardware/sleds/{sled_id}/disks versions = all
25012025-04-30T06:42:45.825ZDEBGtest_snapshot_basic (dropshot_external): registered endpoint local_addr = 127.0.0.1:44640 method = GET path = /v1/system/hardware/sleds/{sled_id}/instances versions = all
25022025-04-30T06:42:45.825ZDEBGtest_snapshot_basic (dropshot_external): registered endpoint local_addr = 127.0.0.1:44640 method = PUT path = /v1/system/hardware/sleds/{sled_id}/provision-policy versions = all
25032025-04-30T06:42:45.825ZDEBGtest_snapshot_basic (dropshot_external): registered endpoint local_addr = 127.0.0.1:44640 method = GET path = /v1/system/hardware/sleds-uninitialized versions = all
25042025-04-30T06:42:45.825ZDEBGtest_snapshot_basic (dropshot_external): registered endpoint local_addr = 127.0.0.1:44640 method = GET path = /v1/system/hardware/switch-port versions = all
25052025-04-30T06:42:45.825ZDEBGtest_snapshot_basic (dropshot_external): registered endpoint local_addr = 127.0.0.1:44640 method = GET path = /v1/system/hardware/switch-port/{port}/lldp/config versions = all
25062025-04-30T06:42:45.825ZDEBGtest_snapshot_basic (dropshot_external): registered endpoint local_addr = 127.0.0.1:44640 method = POST path = /v1/system/hardware/switch-port/{port}/lldp/config versions = all
25072025-04-30T06:42:45.825ZDEBGtest_snapshot_basic (dropshot_external): registered endpoint local_addr = 127.0.0.1:44640 method = DELETE path = /v1/system/hardware/switch-port/{port}/settings versions = all
25082025-04-30T06:42:45.825ZDEBGtest_snapshot_basic (dropshot_external): registered endpoint local_addr = 127.0.0.1:44640 method = POST path = /v1/system/hardware/switch-port/{port}/settings versions = all
25092025-04-30T06:42:45.825ZDEBGtest_snapshot_basic (dropshot_external): registered endpoint local_addr = 127.0.0.1:44640 method = GET path = /v1/system/hardware/switch-port/{port}/status versions = all
25102025-04-30T06:42:45.825ZDEBGtest_snapshot_basic (dropshot_external): registered endpoint local_addr = 127.0.0.1:44640 method = GET path = /v1/system/hardware/switches versions = all
25112025-04-30T06:42:45.825ZDEBGtest_snapshot_basic (dropshot_external): registered endpoint local_addr = 127.0.0.1:44640 method = GET path = /v1/system/hardware/switches/{switch_id} versions = all
25122025-04-30T06:42:45.825ZDEBGtest_snapshot_basic (dropshot_external): registered endpoint local_addr = 127.0.0.1:44640 method = GET path = /v1/system/identity-providers versions = all
25132025-04-30T06:42:45.825ZDEBGtest_snapshot_basic (dropshot_external): registered endpoint local_addr = 127.0.0.1:44640 method = POST path = /v1/system/identity-providers/local/users versions = all
25142025-04-30T06:42:45.825ZDEBGtest_snapshot_basic (dropshot_external): registered endpoint local_addr = 127.0.0.1:44640 method = DELETE path = /v1/system/identity-providers/local/users/{user_id} versions = all
25152025-04-30T06:42:45.825ZDEBGtest_snapshot_basic (dropshot_external): registered endpoint local_addr = 127.0.0.1:44640 method = POST path = /v1/system/identity-providers/local/users/{user_id}/set-password versions = all
25162025-04-30T06:42:45.825ZDEBGtest_snapshot_basic (dropshot_external): registered endpoint local_addr = 127.0.0.1:44640 method = POST path = /v1/system/identity-providers/saml versions = all
25172025-04-30T06:42:45.825ZDEBGtest_snapshot_basic (dropshot_external): registered endpoint local_addr = 127.0.0.1:44640 method = GET path = /v1/system/identity-providers/saml/{provider} versions = all
25182025-04-30T06:42:45.825ZDEBGtest_snapshot_basic (dropshot_external): registered endpoint local_addr = 127.0.0.1:44640 method = GET path = /v1/system/ip-pools versions = all
25192025-04-30T06:42:45.825ZDEBGtest_snapshot_basic (dropshot_external): registered endpoint local_addr = 127.0.0.1:44640 method = POST path = /v1/system/ip-pools versions = all
25202025-04-30T06:42:45.825ZDEBGtest_snapshot_basic (dropshot_external): registered endpoint local_addr = 127.0.0.1:44640 method = DELETE path = /v1/system/ip-pools/{pool} versions = all
25212025-04-30T06:42:45.826ZDEBGtest_snapshot_basic (dropshot_external): registered endpoint local_addr = 127.0.0.1:44640 method = GET path = /v1/system/ip-pools/{pool} versions = all
25222025-04-30T06:42:45.826ZDEBGtest_snapshot_basic (dropshot_external): registered endpoint local_addr = 127.0.0.1:44640 method = PUT path = /v1/system/ip-pools/{pool} versions = all
25232025-04-30T06:42:45.826ZDEBGtest_snapshot_basic (dropshot_external): registered endpoint local_addr = 127.0.0.1:44640 method = GET path = /v1/system/ip-pools/{pool}/ranges versions = all
25242025-04-30T06:42:45.826ZDEBGtest_snapshot_basic (dropshot_external): registered endpoint local_addr = 127.0.0.1:44640 method = POST path = /v1/system/ip-pools/{pool}/ranges/add versions = all
25252025-04-30T06:42:45.826ZDEBGtest_snapshot_basic (dropshot_external): registered endpoint local_addr = 127.0.0.1:44640 method = POST path = /v1/system/ip-pools/{pool}/ranges/remove versions = all
25262025-04-30T06:42:45.826ZDEBGtest_snapshot_basic (dropshot_external): registered endpoint local_addr = 127.0.0.1:44640 method = GET path = /v1/system/ip-pools/{pool}/silos versions = all
25272025-04-30T06:42:45.826ZDEBGtest_snapshot_basic (dropshot_external): registered endpoint local_addr = 127.0.0.1:44640 method = POST path = /v1/system/ip-pools/{pool}/silos versions = all
25282025-04-30T06:42:45.826ZDEBGtest_snapshot_basic (dropshot_external): registered endpoint local_addr = 127.0.0.1:44640 method = DELETE path = /v1/system/ip-pools/{pool}/silos/{silo} versions = all
25292025-04-30T06:42:45.826ZDEBGtest_snapshot_basic (dropshot_external): registered endpoint local_addr = 127.0.0.1:44640 method = PUT path = /v1/system/ip-pools/{pool}/silos/{silo} versions = all
25302025-04-30T06:42:45.826ZDEBGtest_snapshot_basic (dropshot_external): registered endpoint local_addr = 127.0.0.1:44640 method = GET path = /v1/system/ip-pools/{pool}/utilization versions = all
25312025-04-30T06:42:45.826ZDEBGtest_snapshot_basic (dropshot_external): registered endpoint local_addr = 127.0.0.1:44640 method = GET path = /v1/system/ip-pools-service versions = all
25322025-04-30T06:42:45.826ZDEBGtest_snapshot_basic (dropshot_external): registered endpoint local_addr = 127.0.0.1:44640 method = GET path = /v1/system/ip-pools-service/ranges versions = all
25332025-04-30T06:42:45.826ZDEBGtest_snapshot_basic (dropshot_external): registered endpoint local_addr = 127.0.0.1:44640 method = POST path = /v1/system/ip-pools-service/ranges/add versions = all
25342025-04-30T06:42:45.826ZDEBGtest_snapshot_basic (dropshot_external): registered endpoint local_addr = 127.0.0.1:44640 method = POST path = /v1/system/ip-pools-service/ranges/remove versions = all
25352025-04-30T06:42:45.826ZDEBGtest_snapshot_basic (dropshot_external): registered endpoint local_addr = 127.0.0.1:44640 method = GET path = /v1/system/metrics/{metric_name} versions = all
25362025-04-30T06:42:45.826ZDEBGtest_snapshot_basic (dropshot_external): registered endpoint local_addr = 127.0.0.1:44640 method = GET path = /v1/system/networking/address-lot versions = all
25372025-04-30T06:42:45.826ZDEBGtest_snapshot_basic (dropshot_external): registered endpoint local_addr = 127.0.0.1:44640 method = POST path = /v1/system/networking/address-lot versions = all
25382025-04-30T06:42:45.826ZDEBGtest_snapshot_basic (dropshot_external): registered endpoint local_addr = 127.0.0.1:44640 method = DELETE path = /v1/system/networking/address-lot/{address_lot} versions = all
25392025-04-30T06:42:45.826ZDEBGtest_snapshot_basic (dropshot_external): registered endpoint local_addr = 127.0.0.1:44640 method = GET path = /v1/system/networking/address-lot/{address_lot}/blocks versions = all
25402025-04-30T06:42:45.826ZDEBGtest_snapshot_basic (dropshot_external): registered endpoint local_addr = 127.0.0.1:44640 method = GET path = /v1/system/networking/allow-list versions = all
25412025-04-30T06:42:45.826ZDEBGtest_snapshot_basic (dropshot_external): registered endpoint local_addr = 127.0.0.1:44640 method = PUT path = /v1/system/networking/allow-list versions = all
25422025-04-30T06:42:45.826ZDEBGtest_snapshot_basic (dropshot_external): registered endpoint local_addr = 127.0.0.1:44640 method = POST path = /v1/system/networking/bfd-disable versions = all
25432025-04-30T06:42:45.826ZDEBGtest_snapshot_basic (dropshot_external): registered endpoint local_addr = 127.0.0.1:44640 method = POST path = /v1/system/networking/bfd-enable versions = all
25442025-04-30T06:42:45.826ZDEBGtest_snapshot_basic (dropshot_external): registered endpoint local_addr = 127.0.0.1:44640 method = GET path = /v1/system/networking/bfd-status versions = all
25452025-04-30T06:42:45.826ZDEBGtest_snapshot_basic (dropshot_external): registered endpoint local_addr = 127.0.0.1:44640 method = DELETE path = /v1/system/networking/bgp versions = all
25462025-04-30T06:42:45.826ZDEBGtest_snapshot_basic (dropshot_external): registered endpoint local_addr = 127.0.0.1:44640 method = GET path = /v1/system/networking/bgp versions = all
25472025-04-30T06:42:45.826ZDEBGtest_snapshot_basic (dropshot_external): registered endpoint local_addr = 127.0.0.1:44640 method = POST path = /v1/system/networking/bgp versions = all
25482025-04-30T06:42:45.826ZDEBGtest_snapshot_basic (dropshot_external): registered endpoint local_addr = 127.0.0.1:44640 method = GET path = /v1/system/networking/bgp-announce-set versions = all
25492025-04-30T06:42:45.826ZDEBGtest_snapshot_basic (dropshot_external): registered endpoint local_addr = 127.0.0.1:44640 method = PUT path = /v1/system/networking/bgp-announce-set versions = all
25502025-04-30T06:42:45.826ZDEBGtest_snapshot_basic (dropshot_external): registered endpoint local_addr = 127.0.0.1:44640 method = DELETE path = /v1/system/networking/bgp-announce-set/{announce_set} versions = all
25512025-04-30T06:42:45.826ZDEBGtest_snapshot_basic (dropshot_external): registered endpoint local_addr = 127.0.0.1:44640 method = GET path = /v1/system/networking/bgp-announce-set/{announce_set}/announcement versions = all
25522025-04-30T06:42:45.826ZDEBGtest_snapshot_basic (dropshot_external): registered endpoint local_addr = 127.0.0.1:44640 method = GET path = /v1/system/networking/bgp-exported versions = all
25532025-04-30T06:42:45.826ZDEBGtest_snapshot_basic (dropshot_external): registered endpoint local_addr = 127.0.0.1:44640 method = GET path = /v1/system/networking/bgp-message-history versions = all
25542025-04-30T06:42:45.826ZDEBGtest_snapshot_basic (dropshot_external): registered endpoint local_addr = 127.0.0.1:44640 method = GET path = /v1/system/networking/bgp-routes-ipv4 versions = all
25552025-04-30T06:42:45.826ZDEBGtest_snapshot_basic (dropshot_external): registered endpoint local_addr = 127.0.0.1:44640 method = GET path = /v1/system/networking/bgp-status versions = all
25562025-04-30T06:42:45.827ZDEBGtest_snapshot_basic (dropshot_external): registered endpoint local_addr = 127.0.0.1:44640 method = GET path = /v1/system/networking/loopback-address versions = all
25572025-04-30T06:42:45.827ZDEBGtest_snapshot_basic (dropshot_external): registered endpoint local_addr = 127.0.0.1:44640 method = POST path = /v1/system/networking/loopback-address versions = all
25582025-04-30T06:42:45.827ZDEBGtest_snapshot_basic (dropshot_external): registered endpoint local_addr = 127.0.0.1:44640 method = DELETE path = /v1/system/networking/loopback-address/{rack_id}/{switch_location}/{address}/{subnet_mask} versions = all
25592025-04-30T06:42:45.827ZDEBGtest_snapshot_basic (dropshot_external): registered endpoint local_addr = 127.0.0.1:44640 method = DELETE path = /v1/system/networking/switch-port-settings versions = all
25602025-04-30T06:42:45.827ZDEBGtest_snapshot_basic (dropshot_external): registered endpoint local_addr = 127.0.0.1:44640 method = GET path = /v1/system/networking/switch-port-settings versions = all
25612025-04-30T06:42:45.827ZDEBGtest_snapshot_basic (dropshot_external): registered endpoint local_addr = 127.0.0.1:44640 method = POST path = /v1/system/networking/switch-port-settings versions = all
25622025-04-30T06:42:45.827ZDEBGtest_snapshot_basic (dropshot_external): registered endpoint local_addr = 127.0.0.1:44640 method = GET path = /v1/system/networking/switch-port-settings/{port} versions = all
25632025-04-30T06:42:45.827ZDEBGtest_snapshot_basic (dropshot_external): registered endpoint local_addr = 127.0.0.1:44640 method = GET path = /v1/system/policy versions = all
25642025-04-30T06:42:45.827ZDEBGtest_snapshot_basic (dropshot_external): registered endpoint local_addr = 127.0.0.1:44640 method = PUT path = /v1/system/policy versions = all
25652025-04-30T06:42:45.827ZDEBGtest_snapshot_basic (dropshot_external): registered endpoint local_addr = 127.0.0.1:44640 method = GET path = /v1/system/roles versions = all
25662025-04-30T06:42:45.827ZDEBGtest_snapshot_basic (dropshot_external): registered endpoint local_addr = 127.0.0.1:44640 method = GET path = /v1/system/roles/{role_name} versions = all
25672025-04-30T06:42:45.827ZDEBGtest_snapshot_basic (dropshot_external): registered endpoint local_addr = 127.0.0.1:44640 method = GET path = /v1/system/silo-quotas versions = all
25682025-04-30T06:42:45.827ZDEBGtest_snapshot_basic (dropshot_external): registered endpoint local_addr = 127.0.0.1:44640 method = GET path = /v1/system/silos versions = all
25692025-04-30T06:42:45.827ZDEBGtest_snapshot_basic (dropshot_external): registered endpoint local_addr = 127.0.0.1:44640 method = POST path = /v1/system/silos versions = all
25702025-04-30T06:42:45.827ZDEBGtest_snapshot_basic (dropshot_external): registered endpoint local_addr = 127.0.0.1:44640 method = DELETE path = /v1/system/silos/{silo} versions = all
25712025-04-30T06:42:45.827ZDEBGtest_snapshot_basic (dropshot_external): registered endpoint local_addr = 127.0.0.1:44640 method = GET path = /v1/system/silos/{silo} versions = all
25722025-04-30T06:42:45.827ZDEBGtest_snapshot_basic (dropshot_external): registered endpoint local_addr = 127.0.0.1:44640 method = GET path = /v1/system/silos/{silo}/ip-pools versions = all
25732025-04-30T06:42:45.827ZDEBGtest_snapshot_basic (dropshot_external): registered endpoint local_addr = 127.0.0.1:44640 method = GET path = /v1/system/silos/{silo}/policy versions = all
25742025-04-30T06:42:45.827ZDEBGtest_snapshot_basic (dropshot_external): registered endpoint local_addr = 127.0.0.1:44640 method = PUT path = /v1/system/silos/{silo}/policy versions = all
25752025-04-30T06:42:45.827ZDEBGtest_snapshot_basic (dropshot_external): registered endpoint local_addr = 127.0.0.1:44640 method = GET path = /v1/system/silos/{silo}/quotas versions = all
25762025-04-30T06:42:45.827ZDEBGtest_snapshot_basic (dropshot_external): registered endpoint local_addr = 127.0.0.1:44640 method = PUT path = /v1/system/silos/{silo}/quotas versions = all
25772025-04-30T06:42:45.827ZDEBGtest_snapshot_basic (dropshot_external): registered endpoint local_addr = 127.0.0.1:44640 method = POST path = /v1/system/timeseries/query versions = all
25782025-04-30T06:42:45.827ZDEBGtest_snapshot_basic (dropshot_external): registered endpoint local_addr = 127.0.0.1:44640 method = GET path = /v1/system/timeseries/schemas versions = all
25792025-04-30T06:42:45.827ZDEBGtest_snapshot_basic (dropshot_external): registered endpoint local_addr = 127.0.0.1:44640 method = PUT path = /v1/system/update/repository versions = all
25802025-04-30T06:42:45.827ZDEBGtest_snapshot_basic (dropshot_external): registered endpoint local_addr = 127.0.0.1:44640 method = GET path = /v1/system/update/repository/{system_version} versions = all
25812025-04-30T06:42:45.827ZDEBGtest_snapshot_basic (dropshot_external): registered endpoint local_addr = 127.0.0.1:44640 method = GET path = /v1/system/update/target-release versions = all
25822025-04-30T06:42:45.827ZDEBGtest_snapshot_basic (dropshot_external): registered endpoint local_addr = 127.0.0.1:44640 method = PUT path = /v1/system/update/target-release versions = all
25832025-04-30T06:42:45.827ZDEBGtest_snapshot_basic (dropshot_external): registered endpoint local_addr = 127.0.0.1:44640 method = GET path = /v1/system/users versions = all
25842025-04-30T06:42:45.827ZDEBGtest_snapshot_basic (dropshot_external): registered endpoint local_addr = 127.0.0.1:44640 method = GET path = /v1/system/users/{user_id} versions = all
25852025-04-30T06:42:45.827ZDEBGtest_snapshot_basic (dropshot_external): registered endpoint local_addr = 127.0.0.1:44640 method = GET path = /v1/system/users-builtin versions = all
25862025-04-30T06:42:45.827ZDEBGtest_snapshot_basic (dropshot_external): registered endpoint local_addr = 127.0.0.1:44640 method = GET path = /v1/system/users-builtin/{user} versions = all
25872025-04-30T06:42:45.827ZDEBGtest_snapshot_basic (dropshot_external): registered endpoint local_addr = 127.0.0.1:44640 method = GET path = /v1/system/utilization/silos versions = all
25882025-04-30T06:42:45.827ZDEBGtest_snapshot_basic (dropshot_external): registered endpoint local_addr = 127.0.0.1:44640 method = GET path = /v1/system/utilization/silos/{silo} versions = all
25892025-04-30T06:42:45.827ZDEBGtest_snapshot_basic (dropshot_external): registered endpoint local_addr = 127.0.0.1:44640 method = POST path = /v1/timeseries/query versions = all
25902025-04-30T06:42:45.827ZDEBGtest_snapshot_basic (dropshot_external): registered endpoint local_addr = 127.0.0.1:44640 method = GET path = /v1/users versions = all
25912025-04-30T06:42:45.827ZDEBGtest_snapshot_basic (dropshot_external): registered endpoint local_addr = 127.0.0.1:44640 method = GET path = /v1/utilization versions = all
25922025-04-30T06:42:45.827ZDEBGtest_snapshot_basic (dropshot_external): registered endpoint local_addr = 127.0.0.1:44640 method = GET path = /v1/vpc-firewall-rules versions = all
25932025-04-30T06:42:45.827ZDEBGtest_snapshot_basic (dropshot_external): registered endpoint local_addr = 127.0.0.1:44640 method = PUT path = /v1/vpc-firewall-rules versions = all
25942025-04-30T06:42:45.827ZDEBGtest_snapshot_basic (dropshot_external): registered endpoint local_addr = 127.0.0.1:44640 method = GET path = /v1/vpc-router-routes versions = all
25952025-04-30T06:42:45.828ZDEBGtest_snapshot_basic (dropshot_external): registered endpoint local_addr = 127.0.0.1:44640 method = POST path = /v1/vpc-router-routes versions = all
25962025-04-30T06:42:45.828ZDEBGtest_snapshot_basic (dropshot_external): registered endpoint local_addr = 127.0.0.1:44640 method = DELETE path = /v1/vpc-router-routes/{route} versions = all
25972025-04-30T06:42:45.828ZDEBGtest_snapshot_basic (dropshot_external): registered endpoint local_addr = 127.0.0.1:44640 method = GET path = /v1/vpc-router-routes/{route} versions = all
25982025-04-30T06:42:45.828ZDEBGtest_snapshot_basic (dropshot_external): registered endpoint local_addr = 127.0.0.1:44640 method = PUT path = /v1/vpc-router-routes/{route} versions = all
25992025-04-30T06:42:45.828ZDEBGtest_snapshot_basic (dropshot_external): registered endpoint local_addr = 127.0.0.1:44640 method = GET path = /v1/vpc-routers versions = all
26002025-04-30T06:42:45.828ZDEBGtest_snapshot_basic (dropshot_external): registered endpoint local_addr = 127.0.0.1:44640 method = POST path = /v1/vpc-routers versions = all
26012025-04-30T06:42:45.828ZDEBGtest_snapshot_basic (dropshot_external): registered endpoint local_addr = 127.0.0.1:44640 method = DELETE path = /v1/vpc-routers/{router} versions = all
26022025-04-30T06:42:45.828ZDEBGtest_snapshot_basic (dropshot_external): registered endpoint local_addr = 127.0.0.1:44640 method = GET path = /v1/vpc-routers/{router} versions = all
26032025-04-30T06:42:45.828ZDEBGtest_snapshot_basic (dropshot_external): registered endpoint local_addr = 127.0.0.1:44640 method = PUT path = /v1/vpc-routers/{router} versions = all
26042025-04-30T06:42:45.828ZDEBGtest_snapshot_basic (dropshot_external): registered endpoint local_addr = 127.0.0.1:44640 method = GET path = /v1/vpc-subnets versions = all
26052025-04-30T06:42:45.828ZDEBGtest_snapshot_basic (dropshot_external): registered endpoint local_addr = 127.0.0.1:44640 method = POST path = /v1/vpc-subnets versions = all
26062025-04-30T06:42:45.828ZDEBGtest_snapshot_basic (dropshot_external): registered endpoint local_addr = 127.0.0.1:44640 method = DELETE path = /v1/vpc-subnets/{subnet} versions = all
26072025-04-30T06:42:45.828ZDEBGtest_snapshot_basic (dropshot_external): registered endpoint local_addr = 127.0.0.1:44640 method = GET path = /v1/vpc-subnets/{subnet} versions = all
26082025-04-30T06:42:45.828ZDEBGtest_snapshot_basic (dropshot_external): registered endpoint local_addr = 127.0.0.1:44640 method = PUT path = /v1/vpc-subnets/{subnet} versions = all
26092025-04-30T06:42:45.828ZDEBGtest_snapshot_basic (dropshot_external): registered endpoint local_addr = 127.0.0.1:44640 method = GET path = /v1/vpc-subnets/{subnet}/network-interfaces versions = all
26102025-04-30T06:42:45.828ZDEBGtest_snapshot_basic (dropshot_external): registered endpoint local_addr = 127.0.0.1:44640 method = GET path = /v1/vpcs versions = all
26112025-04-30T06:42:45.828ZDEBGtest_snapshot_basic (dropshot_external): registered endpoint local_addr = 127.0.0.1:44640 method = POST path = /v1/vpcs versions = all
26122025-04-30T06:42:45.828ZDEBGtest_snapshot_basic (dropshot_external): registered endpoint local_addr = 127.0.0.1:44640 method = DELETE path = /v1/vpcs/{vpc} versions = all
26132025-04-30T06:42:45.828ZDEBGtest_snapshot_basic (dropshot_external): registered endpoint local_addr = 127.0.0.1:44640 method = GET path = /v1/vpcs/{vpc} versions = all
26142025-04-30T06:42:45.828ZDEBGtest_snapshot_basic (dropshot_external): registered endpoint local_addr = 127.0.0.1:44640 method = PUT path = /v1/vpcs/{vpc} versions = all
26152025-04-30T06:42:45.828ZDEBGtest_snapshot_basic (dropshot_external): registered endpoint local_addr = 127.0.0.1:44640 method = GET path = /v1/webhooks/deliveries versions = all
26162025-04-30T06:42:45.828ZDEBGtest_snapshot_basic (dropshot_external): registered endpoint local_addr = 127.0.0.1:44640 method = POST path = /v1/webhooks/deliveries/{event_id}/resend versions = all
26172025-04-30T06:42:45.828ZDEBGtest_snapshot_basic (dropshot_external): registered endpoint local_addr = 127.0.0.1:44640 method = GET path = /v1/webhooks/event-classes versions = all
26182025-04-30T06:42:45.828ZDEBGtest_snapshot_basic (dropshot_external): registered endpoint local_addr = 127.0.0.1:44640 method = GET path = /v1/webhooks/receivers versions = all
26192025-04-30T06:42:45.828ZDEBGtest_snapshot_basic (dropshot_external): registered endpoint local_addr = 127.0.0.1:44640 method = POST path = /v1/webhooks/receivers versions = all
26202025-04-30T06:42:45.828ZDEBGtest_snapshot_basic (dropshot_external): registered endpoint local_addr = 127.0.0.1:44640 method = DELETE path = /v1/webhooks/receivers/{receiver} versions = all
26212025-04-30T06:42:45.828ZDEBGtest_snapshot_basic (dropshot_external): registered endpoint local_addr = 127.0.0.1:44640 method = GET path = /v1/webhooks/receivers/{receiver} versions = all
26222025-04-30T06:42:45.828ZDEBGtest_snapshot_basic (dropshot_external): registered endpoint local_addr = 127.0.0.1:44640 method = PUT path = /v1/webhooks/receivers/{receiver} versions = all
26232025-04-30T06:42:45.828ZDEBGtest_snapshot_basic (dropshot_external): registered endpoint local_addr = 127.0.0.1:44640 method = POST path = /v1/webhooks/receivers/{receiver}/probe versions = all
26242025-04-30T06:42:45.828ZDEBGtest_snapshot_basic (dropshot_external): registered endpoint local_addr = 127.0.0.1:44640 method = POST path = /v1/webhooks/receivers/{receiver}/subscriptions versions = all
26252025-04-30T06:42:45.828ZDEBGtest_snapshot_basic (dropshot_external): registered endpoint local_addr = 127.0.0.1:44640 method = DELETE path = /v1/webhooks/receivers/{receiver}/subscriptions/{subscription} versions = all
26262025-04-30T06:42:45.828ZDEBGtest_snapshot_basic (dropshot_external): registered endpoint local_addr = 127.0.0.1:44640 method = GET path = /v1/webhooks/secrets versions = all
26272025-04-30T06:42:45.828ZDEBGtest_snapshot_basic (dropshot_external): registered endpoint local_addr = 127.0.0.1:44640 method = POST path = /v1/webhooks/secrets versions = all
26282025-04-30T06:42:45.828ZDEBGtest_snapshot_basic (dropshot_external): registered endpoint local_addr = 127.0.0.1:44640 method = DELETE path = /v1/webhooks/secrets/{secret_id} versions = all
26292025-04-30T06:42:45.828ZINFOtest_snapshot_basic (dropshot_external): listening local_addr = 127.0.0.1:44640
26302025-04-30T06:42:45.828ZDEBGtest_snapshot_basic (dropshot_external): successfully registered DTrace USDT probes local_addr = 127.0.0.1:44640
26312025-04-30T06:42:45.931ZDEBGtest_snapshot_basic (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:41870 method = GET path = / versions = all
26322025-04-30T06:42:45.931ZDEBGtest_snapshot_basic (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:41870 method = GET path = /access versions = all
26332025-04-30T06:42:45.932ZDEBGtest_snapshot_basic (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:41870 method = GET path = /assets/{path} versions = all
26342025-04-30T06:42:45.932ZDEBGtest_snapshot_basic (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:41870 method = POST path = /device/auth versions = all
26352025-04-30T06:42:45.932ZDEBGtest_snapshot_basic (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:41870 method = POST path = /device/confirm versions = all
26362025-04-30T06:42:45.932ZDEBGtest_snapshot_basic (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:41870 method = GET path = /device/success versions = all
26372025-04-30T06:42:45.932ZDEBGtest_snapshot_basic (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:41870 method = POST path = /device/token versions = all
26382025-04-30T06:42:45.932ZDEBGtest_snapshot_basic (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:41870 method = GET path = /device/verify versions = all
26392025-04-30T06:42:45.932ZDEBGtest_snapshot_basic (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:41870 method = GET path = /experimental/v1/probes versions = all
26402025-04-30T06:42:45.932ZDEBGtest_snapshot_basic (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:41870 method = POST path = /experimental/v1/probes versions = all
26412025-04-30T06:42:45.932ZDEBGtest_snapshot_basic (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:41870 method = DELETE path = /experimental/v1/probes/{probe} versions = all
26422025-04-30T06:42:45.932ZDEBGtest_snapshot_basic (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:41870 method = GET path = /experimental/v1/probes/{probe} versions = all
26432025-04-30T06:42:45.932ZDEBGtest_snapshot_basic (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:41870 method = GET path = /experimental/v1/system/support-bundles versions = all
26442025-04-30T06:42:45.932ZDEBGtest_snapshot_basic (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:41870 method = POST path = /experimental/v1/system/support-bundles versions = all
26452025-04-30T06:42:45.932ZDEBGtest_snapshot_basic (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:41870 method = DELETE path = /experimental/v1/system/support-bundles/{support_bundle} versions = all
26462025-04-30T06:42:45.932ZDEBGtest_snapshot_basic (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:41870 method = GET path = /experimental/v1/system/support-bundles/{support_bundle} versions = all
26472025-04-30T06:42:45.932ZDEBGtest_snapshot_basic (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:41870 method = GET path = /experimental/v1/system/support-bundles/{support_bundle}/download versions = all
26482025-04-30T06:42:45.932ZDEBGtest_snapshot_basic (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:41870 method = HEAD path = /experimental/v1/system/support-bundles/{support_bundle}/download versions = all
26492025-04-30T06:42:45.932ZDEBGtest_snapshot_basic (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:41870 method = GET path = /experimental/v1/system/support-bundles/{support_bundle}/download/{file} versions = all
26502025-04-30T06:42:45.932ZDEBGtest_snapshot_basic (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:41870 method = HEAD path = /experimental/v1/system/support-bundles/{support_bundle}/download/{file} versions = all
26512025-04-30T06:42:45.932ZDEBGtest_snapshot_basic (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:41870 method = GET path = /experimental/v1/system/support-bundles/{support_bundle}/index versions = all
26522025-04-30T06:42:45.932ZDEBGtest_snapshot_basic (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:41870 method = GET path = /images versions = all
26532025-04-30T06:42:45.932ZDEBGtest_snapshot_basic (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:41870 method = GET path = /login versions = all
26542025-04-30T06:42:45.932ZDEBGtest_snapshot_basic (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:41870 method = GET path = /login/{silo_name}/local versions = all
26552025-04-30T06:42:45.932ZDEBGtest_snapshot_basic (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:41870 method = GET path = /login/{silo_name}/saml/{provider_name} versions = all
26562025-04-30T06:42:45.932ZDEBGtest_snapshot_basic (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:41870 method = POST path = /login/{silo_name}/saml/{provider_name} versions = all
26572025-04-30T06:42:45.932ZDEBGtest_snapshot_basic (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:41870 method = GET path = /login/{silo_name}/saml/{provider_name}/redirect versions = all
26582025-04-30T06:42:45.932ZDEBGtest_snapshot_basic (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:41870 method = GET path = /lookup/{path} versions = all
26592025-04-30T06:42:45.932ZDEBGtest_snapshot_basic (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:41870 method = GET path = /projects/{path} versions = all
26602025-04-30T06:42:45.932ZDEBGtest_snapshot_basic (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:41870 method = GET path = /projects-new versions = all
26612025-04-30T06:42:45.932ZDEBGtest_snapshot_basic (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:41870 method = GET path = /settings/{path} versions = all
26622025-04-30T06:42:45.932ZDEBGtest_snapshot_basic (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:41870 method = GET path = /system/{path} versions = all
26632025-04-30T06:42:45.932ZDEBGtest_snapshot_basic (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:41870 method = GET path = /utilization versions = all
26642025-04-30T06:42:45.932ZDEBGtest_snapshot_basic (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:41870 method = GET path = /v1/affinity-groups versions = all
26652025-04-30T06:42:45.932ZDEBGtest_snapshot_basic (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:41870 method = POST path = /v1/affinity-groups versions = all
26662025-04-30T06:42:45.933ZDEBGtest_snapshot_basic (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:41870 method = DELETE path = /v1/affinity-groups/{affinity_group} versions = all
26672025-04-30T06:42:45.933ZDEBGtest_snapshot_basic (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:41870 method = GET path = /v1/affinity-groups/{affinity_group} versions = all
26682025-04-30T06:42:45.933ZDEBGtest_snapshot_basic (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:41870 method = PUT path = /v1/affinity-groups/{affinity_group} versions = all
26692025-04-30T06:42:45.933ZDEBGtest_snapshot_basic (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:41870 method = GET path = /v1/affinity-groups/{affinity_group}/members versions = all
26702025-04-30T06:42:45.933ZDEBGtest_snapshot_basic (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:41870 method = DELETE path = /v1/affinity-groups/{affinity_group}/members/instance/{instance} versions = all
26712025-04-30T06:42:45.933ZDEBGtest_snapshot_basic (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:41870 method = GET path = /v1/affinity-groups/{affinity_group}/members/instance/{instance} versions = all
26722025-04-30T06:42:45.933ZDEBGtest_snapshot_basic (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:41870 method = POST path = /v1/affinity-groups/{affinity_group}/members/instance/{instance} versions = all
26732025-04-30T06:42:45.933ZDEBGtest_snapshot_basic (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:41870 method = GET path = /v1/anti-affinity-groups versions = all
26742025-04-30T06:42:45.933ZDEBGtest_snapshot_basic (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:41870 method = POST path = /v1/anti-affinity-groups versions = all
26752025-04-30T06:42:45.933ZDEBGtest_snapshot_basic (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:41870 method = DELETE path = /v1/anti-affinity-groups/{anti_affinity_group} versions = all
26762025-04-30T06:42:45.933ZDEBGtest_snapshot_basic (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:41870 method = GET path = /v1/anti-affinity-groups/{anti_affinity_group} versions = all
26772025-04-30T06:42:45.933ZDEBGtest_snapshot_basic (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:41870 method = PUT path = /v1/anti-affinity-groups/{anti_affinity_group} versions = all
26782025-04-30T06:42:45.933ZDEBGtest_snapshot_basic (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:41870 method = GET path = /v1/anti-affinity-groups/{anti_affinity_group}/members versions = all
26792025-04-30T06:42:45.933ZDEBGtest_snapshot_basic (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:41870 method = DELETE path = /v1/anti-affinity-groups/{anti_affinity_group}/members/instance/{instance} versions = all
26802025-04-30T06:42:45.933ZDEBGtest_snapshot_basic (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:41870 method = GET path = /v1/anti-affinity-groups/{anti_affinity_group}/members/instance/{instance} versions = all
26812025-04-30T06:42:45.933ZDEBGtest_snapshot_basic (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:41870 method = POST path = /v1/anti-affinity-groups/{anti_affinity_group}/members/instance/{instance} versions = all
26822025-04-30T06:42:45.933ZDEBGtest_snapshot_basic (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:41870 method = GET path = /v1/certificates versions = all
26832025-04-30T06:42:45.933ZDEBGtest_snapshot_basic (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:41870 method = POST path = /v1/certificates versions = all
26842025-04-30T06:42:45.933ZDEBGtest_snapshot_basic (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:41870 method = DELETE path = /v1/certificates/{certificate} versions = all
26852025-04-30T06:42:45.933ZDEBGtest_snapshot_basic (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:41870 method = GET path = /v1/certificates/{certificate} versions = all
26862025-04-30T06:42:45.933ZDEBGtest_snapshot_basic (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:41870 method = GET path = /v1/disks versions = all
26872025-04-30T06:42:45.933ZDEBGtest_snapshot_basic (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:41870 method = POST path = /v1/disks versions = all
26882025-04-30T06:42:45.933ZDEBGtest_snapshot_basic (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:41870 method = DELETE path = /v1/disks/{disk} versions = all
26892025-04-30T06:42:45.933ZDEBGtest_snapshot_basic (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:41870 method = GET path = /v1/disks/{disk} versions = all
26902025-04-30T06:42:45.933ZDEBGtest_snapshot_basic (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:41870 method = POST path = /v1/disks/{disk}/bulk-write versions = all
26912025-04-30T06:42:45.933ZDEBGtest_snapshot_basic (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:41870 method = POST path = /v1/disks/{disk}/bulk-write-start versions = all
26922025-04-30T06:42:45.933ZDEBGtest_snapshot_basic (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:41870 method = POST path = /v1/disks/{disk}/bulk-write-stop versions = all
26932025-04-30T06:42:45.933ZDEBGtest_snapshot_basic (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:41870 method = POST path = /v1/disks/{disk}/finalize versions = all
26942025-04-30T06:42:45.933ZDEBGtest_snapshot_basic (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:41870 method = GET path = /v1/disks/{disk}/metrics/{metric} versions = all
26952025-04-30T06:42:45.933ZDEBGtest_snapshot_basic (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:41870 method = GET path = /v1/floating-ips versions = all
26962025-04-30T06:42:45.933ZDEBGtest_snapshot_basic (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:41870 method = POST path = /v1/floating-ips versions = all
26972025-04-30T06:42:45.934ZDEBGtest_snapshot_basic (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:41870 method = DELETE path = /v1/floating-ips/{floating_ip} versions = all
26982025-04-30T06:42:45.934ZDEBGtest_snapshot_basic (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:41870 method = GET path = /v1/floating-ips/{floating_ip} versions = all
26992025-04-30T06:42:45.934ZDEBGtest_snapshot_basic (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:41870 method = PUT path = /v1/floating-ips/{floating_ip} versions = all
27002025-04-30T06:42:45.934ZDEBGtest_snapshot_basic (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:41870 method = POST path = /v1/floating-ips/{floating_ip}/attach versions = all
27012025-04-30T06:42:45.934ZDEBGtest_snapshot_basic (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:41870 method = POST path = /v1/floating-ips/{floating_ip}/detach versions = all
27022025-04-30T06:42:45.934ZDEBGtest_snapshot_basic (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:41870 method = GET path = /v1/groups versions = all
27032025-04-30T06:42:45.934ZDEBGtest_snapshot_basic (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:41870 method = GET path = /v1/groups/{group_id} versions = all
27042025-04-30T06:42:45.934ZDEBGtest_snapshot_basic (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:41870 method = GET path = /v1/images versions = all
27052025-04-30T06:42:45.934ZDEBGtest_snapshot_basic (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:41870 method = POST path = /v1/images versions = all
27062025-04-30T06:42:45.934ZDEBGtest_snapshot_basic (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:41870 method = DELETE path = /v1/images/{image} versions = all
27072025-04-30T06:42:45.934ZDEBGtest_snapshot_basic (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:41870 method = GET path = /v1/images/{image} versions = all
27082025-04-30T06:42:45.934ZDEBGtest_snapshot_basic (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:41870 method = POST path = /v1/images/{image}/demote versions = all
27092025-04-30T06:42:45.934ZDEBGtest_snapshot_basic (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:41870 method = POST path = /v1/images/{image}/promote versions = all
27102025-04-30T06:42:45.934ZDEBGtest_snapshot_basic (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:41870 method = GET path = /v1/instances versions = all
27112025-04-30T06:42:45.934ZDEBGtest_snapshot_basic (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:41870 method = POST path = /v1/instances versions = all
27122025-04-30T06:42:45.934ZDEBGtest_snapshot_basic (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:41870 method = DELETE path = /v1/instances/{instance} versions = all
27132025-04-30T06:42:45.934ZDEBGtest_snapshot_basic (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:41870 method = GET path = /v1/instances/{instance} versions = all
27142025-04-30T06:42:45.934ZDEBGtest_snapshot_basic (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:41870 method = PUT path = /v1/instances/{instance} versions = all
27152025-04-30T06:42:45.934ZDEBGtest_snapshot_basic (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:41870 method = GET path = /v1/instances/{instance}/affinity-groups versions = all
27162025-04-30T06:42:45.934ZDEBGtest_snapshot_basic (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:41870 method = GET path = /v1/instances/{instance}/anti-affinity-groups versions = all
27172025-04-30T06:42:45.934ZDEBGtest_snapshot_basic (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:41870 method = GET path = /v1/instances/{instance}/disks versions = all
27182025-04-30T06:42:45.934ZDEBGtest_snapshot_basic (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:41870 method = POST path = /v1/instances/{instance}/disks/attach versions = all
27192025-04-30T06:42:45.934ZDEBGtest_snapshot_basic (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:41870 method = POST path = /v1/instances/{instance}/disks/detach versions = all
27202025-04-30T06:42:45.934ZDEBGtest_snapshot_basic (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:41870 method = GET path = /v1/instances/{instance}/external-ips versions = all
27212025-04-30T06:42:45.934ZDEBGtest_snapshot_basic (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:41870 method = DELETE path = /v1/instances/{instance}/external-ips/ephemeral versions = all
27222025-04-30T06:42:45.934ZDEBGtest_snapshot_basic (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:41870 method = POST path = /v1/instances/{instance}/external-ips/ephemeral versions = all
27232025-04-30T06:42:45.934ZDEBGtest_snapshot_basic (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:41870 method = POST path = /v1/instances/{instance}/reboot versions = all
27242025-04-30T06:42:45.934ZDEBGtest_snapshot_basic (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:41870 method = GET path = /v1/instances/{instance}/serial-console versions = all
27252025-04-30T06:42:45.934ZDEBGtest_snapshot_basic (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:41870 method = GET path = /v1/instances/{instance}/serial-console/stream versions = all
27262025-04-30T06:42:45.934ZDEBGtest_snapshot_basic (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:41870 method = GET path = /v1/instances/{instance}/ssh-public-keys versions = all
27272025-04-30T06:42:45.934ZDEBGtest_snapshot_basic (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:41870 method = POST path = /v1/instances/{instance}/start versions = all
27282025-04-30T06:42:45.934ZDEBGtest_snapshot_basic (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:41870 method = POST path = /v1/instances/{instance}/stop versions = all
27292025-04-30T06:42:45.934ZDEBGtest_snapshot_basic (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:41870 method = GET path = /v1/internet-gateway-ip-addresses versions = all
27302025-04-30T06:42:45.934ZDEBGtest_snapshot_basic (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:41870 method = POST path = /v1/internet-gateway-ip-addresses versions = all
27312025-04-30T06:42:45.934ZDEBGtest_snapshot_basic (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:41870 method = DELETE path = /v1/internet-gateway-ip-addresses/{address} versions = all
27322025-04-30T06:42:45.934ZDEBGtest_snapshot_basic (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:41870 method = GET path = /v1/internet-gateway-ip-pools versions = all
27332025-04-30T06:42:45.935ZDEBGtest_snapshot_basic (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:41870 method = POST path = /v1/internet-gateway-ip-pools versions = all
27342025-04-30T06:42:45.935ZDEBGtest_snapshot_basic (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:41870 method = DELETE path = /v1/internet-gateway-ip-pools/{pool} versions = all
27352025-04-30T06:42:45.935ZDEBGtest_snapshot_basic (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:41870 method = GET path = /v1/internet-gateways versions = all
27362025-04-30T06:42:45.935ZDEBGtest_snapshot_basic (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:41870 method = POST path = /v1/internet-gateways versions = all
27372025-04-30T06:42:45.935ZDEBGtest_snapshot_basic (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:41870 method = DELETE path = /v1/internet-gateways/{gateway} versions = all
27382025-04-30T06:42:45.935ZDEBGtest_snapshot_basic (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:41870 method = GET path = /v1/internet-gateways/{gateway} versions = all
27392025-04-30T06:42:45.935ZDEBGtest_snapshot_basic (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:41870 method = GET path = /v1/ip-pools versions = all
27402025-04-30T06:42:45.935ZDEBGtest_snapshot_basic (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:41870 method = GET path = /v1/ip-pools/{pool} versions = all
27412025-04-30T06:42:45.935ZDEBGtest_snapshot_basic (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:41870 method = POST path = /v1/login/{silo_name}/local versions = all
27422025-04-30T06:42:45.935ZDEBGtest_snapshot_basic (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:41870 method = POST path = /v1/logout versions = all
27432025-04-30T06:42:45.935ZDEBGtest_snapshot_basic (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:41870 method = GET path = /v1/me versions = all
27442025-04-30T06:42:45.935ZDEBGtest_snapshot_basic (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:41870 method = GET path = /v1/me/groups versions = all
27452025-04-30T06:42:45.935ZDEBGtest_snapshot_basic (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:41870 method = GET path = /v1/me/ssh-keys versions = all
27462025-04-30T06:42:45.935ZDEBGtest_snapshot_basic (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:41870 method = POST path = /v1/me/ssh-keys versions = all
27472025-04-30T06:42:45.935ZDEBGtest_snapshot_basic (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:41870 method = DELETE path = /v1/me/ssh-keys/{ssh_key} versions = all
27482025-04-30T06:42:45.935ZDEBGtest_snapshot_basic (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:41870 method = GET path = /v1/me/ssh-keys/{ssh_key} versions = all
27492025-04-30T06:42:45.935ZDEBGtest_snapshot_basic (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:41870 method = GET path = /v1/metrics/{metric_name} versions = all
27502025-04-30T06:42:45.935ZDEBGtest_snapshot_basic (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:41870 method = GET path = /v1/network-interfaces versions = all
27512025-04-30T06:42:45.935ZDEBGtest_snapshot_basic (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:41870 method = POST path = /v1/network-interfaces versions = all
27522025-04-30T06:42:45.935ZDEBGtest_snapshot_basic (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:41870 method = DELETE path = /v1/network-interfaces/{interface} versions = all
27532025-04-30T06:42:45.935ZDEBGtest_snapshot_basic (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:41870 method = GET path = /v1/network-interfaces/{interface} versions = all
27542025-04-30T06:42:45.935ZDEBGtest_snapshot_basic (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:41870 method = PUT path = /v1/network-interfaces/{interface} versions = all
27552025-04-30T06:42:45.935ZDEBGtest_snapshot_basic (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:41870 method = GET path = /v1/ping versions = all
27562025-04-30T06:42:45.935ZDEBGtest_snapshot_basic (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:41870 method = GET path = /v1/policy versions = all
27572025-04-30T06:42:45.935ZDEBGtest_snapshot_basic (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:41870 method = PUT path = /v1/policy versions = all
27582025-04-30T06:42:45.935ZDEBGtest_snapshot_basic (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:41870 method = GET path = /v1/projects versions = all
27592025-04-30T06:42:45.935ZDEBGtest_snapshot_basic (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:41870 method = POST path = /v1/projects versions = all
27602025-04-30T06:42:45.935ZDEBGtest_snapshot_basic (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:41870 method = DELETE path = /v1/projects/{project} versions = all
27612025-04-30T06:42:45.935ZDEBGtest_snapshot_basic (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:41870 method = GET path = /v1/projects/{project} versions = all
27622025-04-30T06:42:45.935ZDEBGtest_snapshot_basic (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:41870 method = PUT path = /v1/projects/{project} versions = all
27632025-04-30T06:42:45.935ZDEBGtest_snapshot_basic (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:41870 method = GET path = /v1/projects/{project}/policy versions = all
27642025-04-30T06:42:45.935ZDEBGtest_snapshot_basic (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:41870 method = PUT path = /v1/projects/{project}/policy versions = all
27652025-04-30T06:42:45.935ZDEBGtest_snapshot_basic (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:41870 method = GET path = /v1/snapshots versions = all
27662025-04-30T06:42:45.935ZDEBGtest_snapshot_basic (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:41870 method = POST path = /v1/snapshots versions = all
27672025-04-30T06:42:45.936ZDEBGtest_snapshot_basic (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:41870 method = DELETE path = /v1/snapshots/{snapshot} versions = all
27682025-04-30T06:42:45.936ZDEBGtest_snapshot_basic (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:41870 method = GET path = /v1/snapshots/{snapshot} versions = all
27692025-04-30T06:42:45.936ZDEBGtest_snapshot_basic (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:41870 method = GET path = /v1/system/hardware/disks versions = all
27702025-04-30T06:42:45.936ZDEBGtest_snapshot_basic (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:41870 method = GET path = /v1/system/hardware/disks/{disk_id} versions = all
27712025-04-30T06:42:45.936ZDEBGtest_snapshot_basic (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:41870 method = GET path = /v1/system/hardware/rack-switch-port/{rack_id}/{switch_location}/{port}/lldp/neighbors versions = all
27722025-04-30T06:42:45.936ZDEBGtest_snapshot_basic (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:41870 method = GET path = /v1/system/hardware/racks versions = all
27732025-04-30T06:42:45.936ZDEBGtest_snapshot_basic (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:41870 method = GET path = /v1/system/hardware/racks/{rack_id} versions = all
27742025-04-30T06:42:45.936ZDEBGtest_snapshot_basic (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:41870 method = GET path = /v1/system/hardware/sleds versions = all
27752025-04-30T06:42:45.936ZDEBGtest_snapshot_basic (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:41870 method = POST path = /v1/system/hardware/sleds versions = all
27762025-04-30T06:42:45.936ZDEBGtest_snapshot_basic (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:41870 method = GET path = /v1/system/hardware/sleds/{sled_id} versions = all
27772025-04-30T06:42:45.936ZDEBGtest_snapshot_basic (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:41870 method = GET path = /v1/system/hardware/sleds/{sled_id}/disks versions = all
27782025-04-30T06:42:45.936ZDEBGtest_snapshot_basic (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:41870 method = GET path = /v1/system/hardware/sleds/{sled_id}/instances versions = all
27792025-04-30T06:42:45.936ZDEBGtest_snapshot_basic (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:41870 method = PUT path = /v1/system/hardware/sleds/{sled_id}/provision-policy versions = all
27802025-04-30T06:42:45.936ZDEBGtest_snapshot_basic (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:41870 method = GET path = /v1/system/hardware/sleds-uninitialized versions = all
27812025-04-30T06:42:45.936ZDEBGtest_snapshot_basic (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:41870 method = GET path = /v1/system/hardware/switch-port versions = all
27822025-04-30T06:42:45.936ZDEBGtest_snapshot_basic (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:41870 method = GET path = /v1/system/hardware/switch-port/{port}/lldp/config versions = all
27832025-04-30T06:42:45.936ZDEBGtest_snapshot_basic (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:41870 method = POST path = /v1/system/hardware/switch-port/{port}/lldp/config versions = all
27842025-04-30T06:42:45.936ZDEBGtest_snapshot_basic (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:41870 method = DELETE path = /v1/system/hardware/switch-port/{port}/settings versions = all
27852025-04-30T06:42:45.936ZDEBGtest_snapshot_basic (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:41870 method = POST path = /v1/system/hardware/switch-port/{port}/settings versions = all
27862025-04-30T06:42:45.936ZDEBGtest_snapshot_basic (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:41870 method = GET path = /v1/system/hardware/switch-port/{port}/status versions = all
27872025-04-30T06:42:45.936ZDEBGtest_snapshot_basic (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:41870 method = GET path = /v1/system/hardware/switches versions = all
27882025-04-30T06:42:45.936ZDEBGtest_snapshot_basic (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:41870 method = GET path = /v1/system/hardware/switches/{switch_id} versions = all
27892025-04-30T06:42:45.936ZDEBGtest_snapshot_basic (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:41870 method = GET path = /v1/system/identity-providers versions = all
27902025-04-30T06:42:45.936ZDEBGtest_snapshot_basic (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:41870 method = POST path = /v1/system/identity-providers/local/users versions = all
27912025-04-30T06:42:45.936ZDEBGtest_snapshot_basic (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:41870 method = DELETE path = /v1/system/identity-providers/local/users/{user_id} versions = all
27922025-04-30T06:42:45.936ZDEBGtest_snapshot_basic (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:41870 method = POST path = /v1/system/identity-providers/local/users/{user_id}/set-password versions = all
27932025-04-30T06:42:45.936ZDEBGtest_snapshot_basic (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:41870 method = POST path = /v1/system/identity-providers/saml versions = all
27942025-04-30T06:42:45.936ZDEBGtest_snapshot_basic (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:41870 method = GET path = /v1/system/identity-providers/saml/{provider} versions = all
27952025-04-30T06:42:45.936ZDEBGtest_snapshot_basic (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:41870 method = GET path = /v1/system/ip-pools versions = all
27962025-04-30T06:42:45.936ZDEBGtest_snapshot_basic (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:41870 method = POST path = /v1/system/ip-pools versions = all
27972025-04-30T06:42:45.936ZDEBGtest_snapshot_basic (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:41870 method = DELETE path = /v1/system/ip-pools/{pool} versions = all
27982025-04-30T06:42:45.936ZDEBGtest_snapshot_basic (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:41870 method = GET path = /v1/system/ip-pools/{pool} versions = all
27992025-04-30T06:42:45.936ZDEBGtest_snapshot_basic (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:41870 method = PUT path = /v1/system/ip-pools/{pool} versions = all
28002025-04-30T06:42:45.936ZDEBGtest_snapshot_basic (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:41870 method = GET path = /v1/system/ip-pools/{pool}/ranges versions = all
28012025-04-30T06:42:45.936ZDEBGtest_snapshot_basic (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:41870 method = POST path = /v1/system/ip-pools/{pool}/ranges/add versions = all
28022025-04-30T06:42:45.936ZDEBGtest_snapshot_basic (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:41870 method = POST path = /v1/system/ip-pools/{pool}/ranges/remove versions = all
28032025-04-30T06:42:45.937ZDEBGtest_snapshot_basic (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:41870 method = GET path = /v1/system/ip-pools/{pool}/silos versions = all
28042025-04-30T06:42:45.937ZDEBGtest_snapshot_basic (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:41870 method = POST path = /v1/system/ip-pools/{pool}/silos versions = all
28052025-04-30T06:42:45.937ZDEBGtest_snapshot_basic (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:41870 method = DELETE path = /v1/system/ip-pools/{pool}/silos/{silo} versions = all
28062025-04-30T06:42:45.937ZDEBGtest_snapshot_basic (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:41870 method = PUT path = /v1/system/ip-pools/{pool}/silos/{silo} versions = all
28072025-04-30T06:42:45.937ZDEBGtest_snapshot_basic (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:41870 method = GET path = /v1/system/ip-pools/{pool}/utilization versions = all
28082025-04-30T06:42:45.937ZDEBGtest_snapshot_basic (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:41870 method = GET path = /v1/system/ip-pools-service versions = all
28092025-04-30T06:42:45.937ZDEBGtest_snapshot_basic (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:41870 method = GET path = /v1/system/ip-pools-service/ranges versions = all
28102025-04-30T06:42:45.937ZDEBGtest_snapshot_basic (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:41870 method = POST path = /v1/system/ip-pools-service/ranges/add versions = all
28112025-04-30T06:42:45.937ZDEBGtest_snapshot_basic (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:41870 method = POST path = /v1/system/ip-pools-service/ranges/remove versions = all
28122025-04-30T06:42:45.937ZDEBGtest_snapshot_basic (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:41870 method = GET path = /v1/system/metrics/{metric_name} versions = all
28132025-04-30T06:42:45.937ZDEBGtest_snapshot_basic (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:41870 method = GET path = /v1/system/networking/address-lot versions = all
28142025-04-30T06:42:45.937ZDEBGtest_snapshot_basic (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:41870 method = POST path = /v1/system/networking/address-lot versions = all
28152025-04-30T06:42:45.937ZDEBGtest_snapshot_basic (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:41870 method = DELETE path = /v1/system/networking/address-lot/{address_lot} versions = all
28162025-04-30T06:42:45.937ZDEBGtest_snapshot_basic (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:41870 method = GET path = /v1/system/networking/address-lot/{address_lot}/blocks versions = all
28172025-04-30T06:42:45.937ZDEBGtest_snapshot_basic (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:41870 method = GET path = /v1/system/networking/allow-list versions = all
28182025-04-30T06:42:45.937ZDEBGtest_snapshot_basic (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:41870 method = PUT path = /v1/system/networking/allow-list versions = all
28192025-04-30T06:42:45.937ZDEBGtest_snapshot_basic (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:41870 method = POST path = /v1/system/networking/bfd-disable versions = all
28202025-04-30T06:42:45.937ZDEBGtest_snapshot_basic (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:41870 method = POST path = /v1/system/networking/bfd-enable versions = all
28212025-04-30T06:42:45.937ZDEBGtest_snapshot_basic (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:41870 method = GET path = /v1/system/networking/bfd-status versions = all
28222025-04-30T06:42:45.937ZDEBGtest_snapshot_basic (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:41870 method = DELETE path = /v1/system/networking/bgp versions = all
28232025-04-30T06:42:45.937ZDEBGtest_snapshot_basic (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:41870 method = GET path = /v1/system/networking/bgp versions = all
28242025-04-30T06:42:45.937ZDEBGtest_snapshot_basic (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:41870 method = POST path = /v1/system/networking/bgp versions = all
28252025-04-30T06:42:45.937ZDEBGtest_snapshot_basic (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:41870 method = GET path = /v1/system/networking/bgp-announce-set versions = all
28262025-04-30T06:42:45.937ZDEBGtest_snapshot_basic (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:41870 method = PUT path = /v1/system/networking/bgp-announce-set versions = all
28272025-04-30T06:42:45.937ZDEBGtest_snapshot_basic (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:41870 method = DELETE path = /v1/system/networking/bgp-announce-set/{announce_set} versions = all
28282025-04-30T06:42:45.937ZDEBGtest_snapshot_basic (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:41870 method = GET path = /v1/system/networking/bgp-announce-set/{announce_set}/announcement versions = all
28292025-04-30T06:42:45.937ZDEBGtest_snapshot_basic (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:41870 method = GET path = /v1/system/networking/bgp-exported versions = all
28302025-04-30T06:42:45.937ZDEBGtest_snapshot_basic (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:41870 method = GET path = /v1/system/networking/bgp-message-history versions = all
28312025-04-30T06:42:45.937ZDEBGtest_snapshot_basic (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:41870 method = GET path = /v1/system/networking/bgp-routes-ipv4 versions = all
28322025-04-30T06:42:45.937ZDEBGtest_snapshot_basic (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:41870 method = GET path = /v1/system/networking/bgp-status versions = all
28332025-04-30T06:42:45.937ZDEBGtest_snapshot_basic (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:41870 method = GET path = /v1/system/networking/loopback-address versions = all
28342025-04-30T06:42:45.937ZDEBGtest_snapshot_basic (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:41870 method = POST path = /v1/system/networking/loopback-address versions = all
28352025-04-30T06:42:45.937ZDEBGtest_snapshot_basic (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:41870 method = DELETE path = /v1/system/networking/loopback-address/{rack_id}/{switch_location}/{address}/{subnet_mask} versions = all
28362025-04-30T06:42:45.937ZDEBGtest_snapshot_basic (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:41870 method = DELETE path = /v1/system/networking/switch-port-settings versions = all
28372025-04-30T06:42:45.937ZDEBGtest_snapshot_basic (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:41870 method = GET path = /v1/system/networking/switch-port-settings versions = all
28382025-04-30T06:42:45.937ZDEBGtest_snapshot_basic (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:41870 method = POST path = /v1/system/networking/switch-port-settings versions = all
28392025-04-30T06:42:45.938ZDEBGtest_snapshot_basic (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:41870 method = GET path = /v1/system/networking/switch-port-settings/{port} versions = all
28402025-04-30T06:42:45.938ZDEBGtest_snapshot_basic (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:41870 method = GET path = /v1/system/policy versions = all
28412025-04-30T06:42:45.938ZDEBGtest_snapshot_basic (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:41870 method = PUT path = /v1/system/policy versions = all
28422025-04-30T06:42:45.938ZDEBGtest_snapshot_basic (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:41870 method = GET path = /v1/system/roles versions = all
28432025-04-30T06:42:45.938ZDEBGtest_snapshot_basic (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:41870 method = GET path = /v1/system/roles/{role_name} versions = all
28442025-04-30T06:42:45.938ZDEBGtest_snapshot_basic (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:41870 method = GET path = /v1/system/silo-quotas versions = all
28452025-04-30T06:42:45.938ZDEBGtest_snapshot_basic (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:41870 method = GET path = /v1/system/silos versions = all
28462025-04-30T06:42:45.938ZDEBGtest_snapshot_basic (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:41870 method = POST path = /v1/system/silos versions = all
28472025-04-30T06:42:45.938ZDEBGtest_snapshot_basic (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:41870 method = DELETE path = /v1/system/silos/{silo} versions = all
28482025-04-30T06:42:45.938ZDEBGtest_snapshot_basic (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:41870 method = GET path = /v1/system/silos/{silo} versions = all
28492025-04-30T06:42:45.938ZDEBGtest_snapshot_basic (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:41870 method = GET path = /v1/system/silos/{silo}/ip-pools versions = all
28502025-04-30T06:42:45.938ZDEBGtest_snapshot_basic (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:41870 method = GET path = /v1/system/silos/{silo}/policy versions = all
28512025-04-30T06:42:45.938ZDEBGtest_snapshot_basic (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:41870 method = PUT path = /v1/system/silos/{silo}/policy versions = all
28522025-04-30T06:42:45.938ZDEBGtest_snapshot_basic (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:41870 method = GET path = /v1/system/silos/{silo}/quotas versions = all
28532025-04-30T06:42:45.938ZDEBGtest_snapshot_basic (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:41870 method = PUT path = /v1/system/silos/{silo}/quotas versions = all
28542025-04-30T06:42:45.938ZDEBGtest_snapshot_basic (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:41870 method = POST path = /v1/system/timeseries/query versions = all
28552025-04-30T06:42:45.938ZDEBGtest_snapshot_basic (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:41870 method = GET path = /v1/system/timeseries/schemas versions = all
28562025-04-30T06:42:45.938ZDEBGtest_snapshot_basic (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:41870 method = PUT path = /v1/system/update/repository versions = all
28572025-04-30T06:42:45.938ZDEBGtest_snapshot_basic (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:41870 method = GET path = /v1/system/update/repository/{system_version} versions = all
28582025-04-30T06:42:45.938ZDEBGtest_snapshot_basic (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:41870 method = GET path = /v1/system/update/target-release versions = all
28592025-04-30T06:42:45.938ZDEBGtest_snapshot_basic (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:41870 method = PUT path = /v1/system/update/target-release versions = all
28602025-04-30T06:42:45.938ZDEBGtest_snapshot_basic (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:41870 method = GET path = /v1/system/users versions = all
28612025-04-30T06:42:45.938ZDEBGtest_snapshot_basic (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:41870 method = GET path = /v1/system/users/{user_id} versions = all
28622025-04-30T06:42:45.938ZDEBGtest_snapshot_basic (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:41870 method = GET path = /v1/system/users-builtin versions = all
28632025-04-30T06:42:45.938ZDEBGtest_snapshot_basic (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:41870 method = GET path = /v1/system/users-builtin/{user} versions = all
28642025-04-30T06:42:45.938ZDEBGtest_snapshot_basic (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:41870 method = GET path = /v1/system/utilization/silos versions = all
28652025-04-30T06:42:45.938ZDEBGtest_snapshot_basic (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:41870 method = GET path = /v1/system/utilization/silos/{silo} versions = all
28662025-04-30T06:42:45.938ZDEBGtest_snapshot_basic (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:41870 method = POST path = /v1/timeseries/query versions = all
28672025-04-30T06:42:45.938ZDEBGtest_snapshot_basic (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:41870 method = GET path = /v1/users versions = all
28682025-04-30T06:42:45.938ZDEBGtest_snapshot_basic (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:41870 method = GET path = /v1/utilization versions = all
28692025-04-30T06:42:45.938ZDEBGtest_snapshot_basic (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:41870 method = GET path = /v1/vpc-firewall-rules versions = all
28702025-04-30T06:42:45.938ZDEBGtest_snapshot_basic (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:41870 method = PUT path = /v1/vpc-firewall-rules versions = all
28712025-04-30T06:42:45.938ZDEBGtest_snapshot_basic (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:41870 method = GET path = /v1/vpc-router-routes versions = all
28722025-04-30T06:42:45.938ZDEBGtest_snapshot_basic (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:41870 method = POST path = /v1/vpc-router-routes versions = all
28732025-04-30T06:42:45.938ZDEBGtest_snapshot_basic (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:41870 method = DELETE path = /v1/vpc-router-routes/{route} versions = all
28742025-04-30T06:42:45.938ZDEBGtest_snapshot_basic (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:41870 method = GET path = /v1/vpc-router-routes/{route} versions = all
28752025-04-30T06:42:45.939ZDEBGtest_snapshot_basic (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:41870 method = PUT path = /v1/vpc-router-routes/{route} versions = all
28762025-04-30T06:42:45.939ZDEBGtest_snapshot_basic (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:41870 method = GET path = /v1/vpc-routers versions = all
28772025-04-30T06:42:45.939ZDEBGtest_snapshot_basic (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:41870 method = POST path = /v1/vpc-routers versions = all
28782025-04-30T06:42:45.939ZDEBGtest_snapshot_basic (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:41870 method = DELETE path = /v1/vpc-routers/{router} versions = all
28792025-04-30T06:42:45.939ZDEBGtest_snapshot_basic (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:41870 method = GET path = /v1/vpc-routers/{router} versions = all
28802025-04-30T06:42:45.939ZDEBGtest_snapshot_basic (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:41870 method = PUT path = /v1/vpc-routers/{router} versions = all
28812025-04-30T06:42:45.939ZDEBGtest_snapshot_basic (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:41870 method = GET path = /v1/vpc-subnets versions = all
28822025-04-30T06:42:45.939ZDEBGtest_snapshot_basic (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:41870 method = POST path = /v1/vpc-subnets versions = all
28832025-04-30T06:42:45.939ZDEBGtest_snapshot_basic (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:41870 method = DELETE path = /v1/vpc-subnets/{subnet} versions = all
28842025-04-30T06:42:45.939ZDEBGtest_snapshot_basic (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:41870 method = GET path = /v1/vpc-subnets/{subnet} versions = all
28852025-04-30T06:42:45.939ZDEBGtest_snapshot_basic (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:41870 method = PUT path = /v1/vpc-subnets/{subnet} versions = all
28862025-04-30T06:42:45.939ZDEBGtest_snapshot_basic (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:41870 method = GET path = /v1/vpc-subnets/{subnet}/network-interfaces versions = all
28872025-04-30T06:42:45.939ZDEBGtest_snapshot_basic (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:41870 method = GET path = /v1/vpcs versions = all
28882025-04-30T06:42:45.939ZDEBGtest_snapshot_basic (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:41870 method = POST path = /v1/vpcs versions = all
28892025-04-30T06:42:45.939ZDEBGtest_snapshot_basic (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:41870 method = DELETE path = /v1/vpcs/{vpc} versions = all
28902025-04-30T06:42:45.939ZDEBGtest_snapshot_basic (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:41870 method = GET path = /v1/vpcs/{vpc} versions = all
28912025-04-30T06:42:45.939ZDEBGtest_snapshot_basic (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:41870 method = PUT path = /v1/vpcs/{vpc} versions = all
28922025-04-30T06:42:45.939ZDEBGtest_snapshot_basic (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:41870 method = GET path = /v1/webhooks/deliveries versions = all
28932025-04-30T06:42:45.939ZDEBGtest_snapshot_basic (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:41870 method = POST path = /v1/webhooks/deliveries/{event_id}/resend versions = all
28942025-04-30T06:42:45.939ZDEBGtest_snapshot_basic (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:41870 method = GET path = /v1/webhooks/event-classes versions = all
28952025-04-30T06:42:45.939ZDEBGtest_snapshot_basic (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:41870 method = GET path = /v1/webhooks/receivers versions = all
28962025-04-30T06:42:45.939ZDEBGtest_snapshot_basic (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:41870 method = POST path = /v1/webhooks/receivers versions = all
28972025-04-30T06:42:45.939ZDEBGtest_snapshot_basic (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:41870 method = DELETE path = /v1/webhooks/receivers/{receiver} versions = all
28982025-04-30T06:42:45.939ZDEBGtest_snapshot_basic (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:41870 method = GET path = /v1/webhooks/receivers/{receiver} versions = all
28992025-04-30T06:42:45.939ZDEBGtest_snapshot_basic (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:41870 method = PUT path = /v1/webhooks/receivers/{receiver} versions = all
29002025-04-30T06:42:45.939ZDEBGtest_snapshot_basic (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:41870 method = POST path = /v1/webhooks/receivers/{receiver}/probe versions = all
29012025-04-30T06:42:45.939ZDEBGtest_snapshot_basic (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:41870 method = POST path = /v1/webhooks/receivers/{receiver}/subscriptions versions = all
29022025-04-30T06:42:45.939ZDEBGtest_snapshot_basic (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:41870 method = DELETE path = /v1/webhooks/receivers/{receiver}/subscriptions/{subscription} versions = all
29032025-04-30T06:42:45.939ZDEBGtest_snapshot_basic (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:41870 method = GET path = /v1/webhooks/secrets versions = all
29042025-04-30T06:42:45.939ZDEBGtest_snapshot_basic (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:41870 method = POST path = /v1/webhooks/secrets versions = all
29052025-04-30T06:42:45.939ZDEBGtest_snapshot_basic (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:41870 method = DELETE path = /v1/webhooks/secrets/{secret_id} versions = all
29062025-04-30T06:42:45.939ZINFOtest_snapshot_basic (dropshot_external_techport): listening local_addr = 127.0.0.1:41870
29072025-04-30T06:42:45.939ZDEBGtest_snapshot_basic (dropshot_external_techport): successfully registered DTrace USDT probes local_addr = 127.0.0.1:41870
29082025-04-30T06:42:45.939ZDEBGtest_snapshot_basic (nexus-producer-server): registered DTrace probes
29092025-04-30T06:42:45.939ZDEBGtest_snapshot_basic (dropshot): registered endpoint local_addr = 127.0.0.1:47319 method = GET path = /{producer_id} versions = all
29102025-04-30T06:42:45.939ZINFOtest_snapshot_basic (dropshot): listening file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:328 local_addr = 127.0.0.1:47319
29112025-04-30T06:42:45.939ZDEBGtest_snapshot_basic (dropshot): successfully registered DTrace USDT probes local_addr = 127.0.0.1:47319
29122025-04-30T06:42:45.939ZDEBGtest_snapshot_basic (nexus-producer-server): Requested any available port, Dropshot server has been bound to 127.0.0.1:47319
29132025-04-30T06:42:45.939ZDEBGtest_snapshot_basic (nexus-producer-server): Nexus IP provided explicitly, will use it addr = 127.0.0.1:61468
29142025-04-30T06:42:45.940ZDEBGtest_snapshot_basic (nexus-producer-server): starting producer registration task
29152025-04-30T06:42:45.940ZINFOtest_snapshot_basic (nexus-producer-server): starting oximeter metric producer server address = 127.0.0.1:47319 file = oximeter/producer/src/lib.rs:279 interval = 10s producer_id = 354a52d4-3078-4706-81ce-12c32f6bbeaa
29162025-04-30T06:42:45.940ZDEBGtest_snapshot_basic (init_with_steps): Running step start_oximeter
29172025-04-30T06:42:45.940ZDEBGtest_snapshot_basic: Starting Oximeter
29182025-04-30T06:42:45.940ZDEBGtest_snapshot_basic (oximeter): registered DTrace probes
29192025-04-30T06:42:45.940ZINFOtest_snapshot_basic (oximeter): starting oximeter server file = oximeter/collector/src/lib.rs:236
29202025-04-30T06:42:45.940ZDEBGtest_snapshot_basic (oximeter): creating ClickHouse client
29212025-04-30T06:42:45.940ZDEBGtest_snapshot_basic (clickhouse-client): registered USDT probes collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 6ad06c6b-9a6e-4ed1-b3f4-5d2e9da3f8a7
29222025-04-30T06:42:45.940ZDEBGtest_snapshot_basic (producer-registration-task): registering as oximeter producer with Nexus registration_count = 0
29232025-04-30T06:42:45.940ZDEBGtest_snapshot_basic (producer-registration-task): will register with Nexus at 127.0.0.1:61468
29242025-04-30T06:42:46.046ZDEBGtest_snapshot_basic (producer-registration-task): client request body = Some(Body) method = POST uri = http://127.0.0.1:61468/metrics/producers
29252025-04-30T06:42:46.050ZINFOtest_snapshot_basic (dropshot_internal): accepted connection local_addr = 127.0.0.1:61468 remote_addr = 127.0.0.1:55851
29262025-04-30T06:42:46.051ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "e1f56726-e3f5-405a-974b-9e8eb14e9f4e", "content-length": "178", "date": "Wed, 30 Apr 2025 06:42:46 GMT"} })
29272025-04-30T06:42:46.051ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/1/component/stage0/caboose?firmware_slot=1
29282025-04-30T06:42:46.057ZTRACtest_snapshot_basic (dropshot_internal): incoming request local_addr = 127.0.0.1:61468 method = POST remote_addr = 127.0.0.1:55851 req_id = fcfc025d-75e0-4863-b85e-ae785b922abf uri = /metrics/producers
29292025-04-30T06:42:46.058ZTRACtest_snapshot_basic (dropshot_internal): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:61468 method = POST remote_addr = 127.0.0.1:55851 req_id = fcfc025d-75e0-4863-b85e-ae785b922abf resource = Database uri = /metrics/producers
29302025-04-30T06:42:46.058ZDEBGtest_snapshot_basic (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:61468 method = POST remote_addr = 127.0.0.1:55851 req_id = fcfc025d-75e0-4863-b85e-ae785b922abf roles = RoleSet { roles: {} } uri = /metrics/producers
29312025-04-30T06:42:46.059ZDEBGtest_snapshot_basic (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:61468 method = POST remote_addr = 127.0.0.1:55851 req_id = fcfc025d-75e0-4863-b85e-ae785b922abf resource = Database result = Ok(()) uri = /metrics/producers
29322025-04-30T06:42:46.061ZTRACtest_snapshot_basic (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 6ad06c6b-9a6e-4ed1-b3f4-5d2e9da3f8a7 sql = SELECT MAX(value) AS max_version FROM oximeter.version;
29332025-04-30T06:42:46.072ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "dd9f2301-a59e-40af-a62d-f0373c203414", "content-length": "178", "date": "Wed, 30 Apr 2025 06:42:46 GMT"} })
29342025-04-30T06:42:46.073ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/1/component/rot/cmpa
29352025-04-30T06:42:46.075ZWARNtest_snapshot_basic (clickhouse-client): oximeter database does not exist, or is out-of-date collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 file = oximeter/db/src/client/mod.rs:894 id = 6ad06c6b-9a6e-4ed1-b3f4-5d2e9da3f8a7
29362025-04-30T06:42:46.075ZDEBGtest_snapshot_basic (oximeter-agent): oximeter database does not exist, creating collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
29372025-04-30T06:42:46.075ZINFOtest_snapshot_basic (clickhouse-client): reading db version collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 file = oximeter/db/src/client/mod.rs:794 id = 6ad06c6b-9a6e-4ed1-b3f4-5d2e9da3f8a7
29382025-04-30T06:42:46.075ZTRACtest_snapshot_basic (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 6ad06c6b-9a6e-4ed1-b3f4-5d2e9da3f8a7 sql = SELECT MAX(value) AS max_version FROM oximeter.version;
29392025-04-30T06:42:46.077ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "e3a2883a-cf2a-4373-878c-88e3d1645796", "content-length": "702", "date": "Wed, 30 Apr 2025 06:42:46 GMT"} })
29402025-04-30T06:42:46.077ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/1/component/rot/cfpa
29412025-04-30T06:42:46.078ZWARNtest_snapshot_basic (clickhouse-client): oximeter database does not exist, or is out-of-date collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 file = oximeter/db/src/client/mod.rs:894 id = 6ad06c6b-9a6e-4ed1-b3f4-5d2e9da3f8a7
29422025-04-30T06:42:46.078ZINFOtest_snapshot_basic (clickhouse-client): read oximeter database version collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 file = oximeter/db/src/client/mod.rs:799 id = 6ad06c6b-9a6e-4ed1-b3f4-5d2e9da3f8a7 version = 0
29432025-04-30T06:42:46.078ZINFOtest_snapshot_basic (clickhouse-client): initializing oximeter schema collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 file = oximeter/db/src/client/mod.rs:804 id = 6ad06c6b-9a6e-4ed1-b3f4-5d2e9da3f8a7
29442025-04-30T06:42:46.078ZDEBGtest_snapshot_basic (clickhouse-client): initializing ClickHouse database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 6ad06c6b-9a6e-4ed1-b3f4-5d2e9da3f8a7
29452025-04-30T06:42:46.078ZTRACtest_snapshot_basic (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 6ad06c6b-9a6e-4ed1-b3f4-5d2e9da3f8a7 sql = CREATE DATABASE IF NOT EXISTS oximeter
29462025-04-30T06:42:46.079ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "4d1e3d89-4d46-4c79-ba3a-3b35afadd56e", "content-length": "727", "date": "Wed, 30 Apr 2025 06:42:46 GMT"} })
29472025-04-30T06:42:46.079ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/1/component/rot/cfpa
29482025-04-30T06:42:46.080ZTRACtest_snapshot_basic (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 6ad06c6b-9a6e-4ed1-b3f4-5d2e9da3f8a7 sql = \n\n/* The version table contains metadata about the `oximeter` database */\nCREATE TABLE IF NOT EXISTS oximeter.version\n(\n value UInt64,\n timestamp DateTime64(9, 'UTC')\n)\nENGINE = MergeTree()\nORDER BY (value, timestamp)
29492025-04-30T06:42:46.080ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "fb12042e-4ff4-4c74-a968-9637fce31000", "content-length": "729", "date": "Wed, 30 Apr 2025 06:42:46 GMT"} })
29502025-04-30T06:42:46.080ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/1/component/rot/cfpa
29512025-04-30T06:42:46.081ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "1337802d-b871-4158-bb41-6021e3ef05a7", "content-length": "728", "date": "Wed, 30 Apr 2025 06:42:46 GMT"} })
29522025-04-30T06:42:46.082ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
29532025-04-30T06:42:46.082ZTRACtest_snapshot_basic (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
29542025-04-30T06:42:46.082ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
29552025-04-30T06:42:46.082ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
29562025-04-30T06:42:46.083ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
29572025-04-30T06:42:46.083ZTRACtest_snapshot_basic (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 6ad06c6b-9a6e-4ed1-b3f4-5d2e9da3f8a7 sql = \n\n/* The measurement tables contain all individual samples from each timeseries.\n *\n * Each table stores a single datum type, and otherwise contains nearly the same\n * structure. The primary sorting key is on the timeseries name, key, and then\n * timestamp, so that all timeseries from the same schema are grouped, followed\n * by all samples from the same timeseries.\n *\n * This reflects that one usually looks up the _key_ in one or more field table,\n * and then uses that to index quickly into the measurements tables.\n */\nCREATE TABLE IF NOT EXISTS oximeter.measurements_bool\n(\n timeseries_name String,\n timeseries_key UInt64,\n timestamp DateTime64(9, 'UTC'),\n datum Nullable(Bool)\n)\nENGINE = MergeTree()\nORDER BY (timeseries_name, timeseries_key, timestamp)\nTTL toDateTime(timestamp) + INTERVAL 30 DAY
29582025-04-30T06:42:46.086ZTRACtest_snapshot_basic (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 6ad06c6b-9a6e-4ed1-b3f4-5d2e9da3f8a7 sql = \n\nCREATE TABLE IF NOT EXISTS oximeter.measurements_i8\n(\n timeseries_name String,\n timeseries_key UInt64,\n timestamp DateTime64(9, 'UTC'),\n datum Nullable(Int8)\n)\nENGINE = MergeTree()\nORDER BY (timeseries_name, timeseries_key, timestamp)\nTTL toDateTime(timestamp) + INTERVAL 30 DAY
29592025-04-30T06:42:46.086ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
29602025-04-30T06:42:46.097ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
29612025-04-30T06:42:46.097ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
29622025-04-30T06:42:46.097ZTRACtest_snapshot_basic (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
29632025-04-30T06:42:46.097ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
29642025-04-30T06:42:46.097ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
29652025-04-30T06:42:46.097ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
29662025-04-30T06:42:46.100ZTRACtest_snapshot_basic (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 6ad06c6b-9a6e-4ed1-b3f4-5d2e9da3f8a7 sql = \n\nCREATE TABLE IF NOT EXISTS oximeter.measurements_u8\n(\n timeseries_name String,\n timeseries_key UInt64,\n timestamp DateTime64(9, 'UTC'),\n datum Nullable(UInt8)\n)\nENGINE = MergeTree()\nORDER BY (timeseries_name, timeseries_key, timestamp)\nTTL toDateTime(timestamp) + INTERVAL 30 DAY
29672025-04-30T06:42:46.103ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
29682025-04-30T06:42:46.116ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
29692025-04-30T06:42:46.116ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
29702025-04-30T06:42:46.116ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
29712025-04-30T06:42:46.117ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
29722025-04-30T06:42:46.117ZINFOtest_snapshot_basic (dropshot_internal): request completed error_message_external = Service Unavailable error_message_internal = no Oximeter instances available for assignment latency_us = 60827 local_addr = 127.0.0.1:61468 method = POST remote_addr = 127.0.0.1:55851 req_id = fcfc025d-75e0-4863-b85e-ae785b922abf response_code = 503 uri = /metrics/producers
29732025-04-30T06:42:46.121ZTRACtest_snapshot_basic (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 6ad06c6b-9a6e-4ed1-b3f4-5d2e9da3f8a7 sql = \n\nCREATE TABLE IF NOT EXISTS oximeter.measurements_i16\n(\n timeseries_name String,\n timeseries_key UInt64,\n timestamp DateTime64(9, 'UTC'),\n datum Nullable(Int16)\n)\nENGINE = MergeTree()\nORDER BY (timeseries_name, timeseries_key, timestamp)\nTTL toDateTime(timestamp) + INTERVAL 30 DAY
29742025-04-30T06:42:46.124ZDEBGtest_snapshot_basic (producer-registration-task): client response result = Ok(Response { url: "http://127.0.0.1:61468/metrics/producers", status: 503, headers: {"content-type": "application/json", "x-request-id": "fcfc025d-75e0-4863-b85e-ae785b922abf", "content-length": "133", "date": "Wed, 30 Apr 2025 06:42:46 GMT"} })
29752025-04-30T06:42:46.126ZWARNtest_snapshot_basic (producer-registration-task): failed to register with Nexus, will retry count = 0 delay = 186.653996ms error = "Error Response: status: 503 Service Unavailable; headers: {\\"content-type\\": \\"application/json\\", \\"x-request-id\\": \\"fcfc025d-75e0-4863-b85e-ae785b922abf\\", \\"content-length\\": \\"133\\", \\"date\\": \\"Wed, 30 Apr 2025 06:42:46 GMT\\"}; value: Error { error_code: Some(\\"ServiceNotAvailable\\"), message: \\"Service Unavailable\\", request_id: \\"fcfc025d-75e0-4863-b85e-ae785b922abf\\" }" file = oximeter/producer/src/lib.rs:371
29762025-04-30T06:42:46.185ZDEBGtest_snapshot_basic (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:41923
29772025-04-30T06:42:46.185ZDEBGtest_snapshot_basic (BackgroundTasks): client request SledAgent = http://[::1]:41923 background_task = inventory_collection body = None method = GET uri = http://[::1]:41923/inventory
29782025-04-30T06:42:46.186ZTRACtest_snapshot_basic (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 6ad06c6b-9a6e-4ed1-b3f4-5d2e9da3f8a7 sql = \n\nCREATE TABLE IF NOT EXISTS oximeter.measurements_u16\n(\n timeseries_name String,\n timeseries_key UInt64,\n timestamp DateTime64(9, 'UTC'),\n datum Nullable(UInt16)\n)\nENGINE = MergeTree()\nORDER BY (timeseries_name, timeseries_key, timestamp)\nTTL toDateTime(timestamp) + INTERVAL 30 DAY
29792025-04-30T06:42:46.189ZINFOtest_snapshot_basic (dropshot): accepted connection local_addr = [::1]:41923 remote_addr = [::1]:41443 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
29802025-04-30T06:42:46.190ZTRACtest_snapshot_basic (dropshot): incoming request local_addr = [::1]:41923 method = GET remote_addr = [::1]:41443 req_id = 4a5309ec-c8d7-47b0-84d6-3ecdecca0f2c sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
29812025-04-30T06:42:46.190ZINFOtest_snapshot_basic (dropshot): request completed latency_us = 582 local_addr = [::1]:41923 method = GET remote_addr = [::1]:41443 req_id = 4a5309ec-c8d7-47b0-84d6-3ecdecca0f2c response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
29822025-04-30T06:42:46.190ZTRACtest_snapshot_basic (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 6ad06c6b-9a6e-4ed1-b3f4-5d2e9da3f8a7 sql = \n\nCREATE TABLE IF NOT EXISTS oximeter.measurements_i32\n(\n timeseries_name String,\n timeseries_key UInt64,\n timestamp DateTime64(9, 'UTC'),\n datum Nullable(Int32)\n)\nENGINE = MergeTree()\nORDER BY (timeseries_name, timeseries_key, timestamp)\nTTL toDateTime(timestamp) + INTERVAL 30 DAY
29832025-04-30T06:42:46.192ZDEBGtest_snapshot_basic (BackgroundTasks): client response SledAgent = http://[::1]:41923 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:41923/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "4a5309ec-c8d7-47b0-84d6-3ecdecca0f2c", "content-length": "2769", "date": "Wed, 30 Apr 2025 06:42:46 GMT"} })
29842025-04-30T06:42:46.194ZDEBGtest_snapshot_basic (BackgroundTasks): begin collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
29852025-04-30T06:42:46.194ZDEBGtest_snapshot_basic (BackgroundTasks): end collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
29862025-04-30T06:42:46.194ZDEBGtest_snapshot_basic (BackgroundTasks): finished collection background_task = inventory_collection
29872025-04-30T06:42:46.196ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
29882025-04-30T06:42:46.196ZTRACtest_snapshot_basic (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
29892025-04-30T06:42:46.196ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
29902025-04-30T06:42:46.196ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
29912025-04-30T06:42:46.197ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
29922025-04-30T06:42:46.199ZTRACtest_snapshot_basic (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 6ad06c6b-9a6e-4ed1-b3f4-5d2e9da3f8a7 sql = \n\nCREATE TABLE IF NOT EXISTS oximeter.measurements_u32\n(\n timeseries_name String,\n timeseries_key UInt64,\n timestamp DateTime64(9, 'UTC'),\n datum Nullable(UInt32)\n)\nENGINE = MergeTree()\nORDER BY (timeseries_name, timeseries_key, timestamp)\nTTL toDateTime(timestamp) + INTERVAL 30 DAY
29932025-04-30T06:42:46.202ZTRACtest_snapshot_basic (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 6ad06c6b-9a6e-4ed1-b3f4-5d2e9da3f8a7 sql = \n\nCREATE TABLE IF NOT EXISTS oximeter.measurements_i64\n(\n timeseries_name String,\n timeseries_key UInt64,\n timestamp DateTime64(9, 'UTC'),\n datum Nullable(Int64)\n)\nENGINE = MergeTree()\nORDER BY (timeseries_name, timeseries_key, timestamp)\nTTL toDateTime(timestamp) + INTERVAL 30 DAY
29942025-04-30T06:42:46.202ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
29952025-04-30T06:42:46.208ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
29962025-04-30T06:42:46.208ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
29972025-04-30T06:42:46.209ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
29982025-04-30T06:42:46.210ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
29992025-04-30T06:42:46.211ZTRACtest_snapshot_basic (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 6ad06c6b-9a6e-4ed1-b3f4-5d2e9da3f8a7 sql = \n\nCREATE TABLE IF NOT EXISTS oximeter.measurements_u64\n(\n timeseries_name String,\n timeseries_key UInt64,\n timestamp DateTime64(9, 'UTC'),\n datum Nullable(UInt64)\n)\nENGINE = MergeTree()\nORDER BY (timeseries_name, timeseries_key, timestamp)\nTTL toDateTime(timestamp) + INTERVAL 30 DAY
30002025-04-30T06:42:46.214ZTRACtest_snapshot_basic (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 6ad06c6b-9a6e-4ed1-b3f4-5d2e9da3f8a7 sql = \n\nCREATE TABLE IF NOT EXISTS oximeter.measurements_f32\n(\n timeseries_name String,\n timeseries_key UInt64,\n timestamp DateTime64(9, 'UTC'),\n datum Nullable(Float32)\n)\nENGINE = MergeTree()\nORDER BY (timeseries_name, timeseries_key, timestamp)\nTTL toDateTime(timestamp) + INTERVAL 30 DAY
30012025-04-30T06:42:46.217ZTRACtest_snapshot_basic (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 6ad06c6b-9a6e-4ed1-b3f4-5d2e9da3f8a7 sql = \n\nCREATE TABLE IF NOT EXISTS oximeter.measurements_f64\n(\n timeseries_name String,\n timeseries_key UInt64,\n timestamp DateTime64(9, 'UTC'),\n datum Nullable(Float64)\n)\nENGINE = MergeTree()\nORDER BY (timeseries_name, timeseries_key, timestamp)\nTTL toDateTime(timestamp) + INTERVAL 30 DAY
30022025-04-30T06:42:46.221ZTRACtest_snapshot_basic (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 6ad06c6b-9a6e-4ed1-b3f4-5d2e9da3f8a7 sql = \n\nCREATE TABLE IF NOT EXISTS oximeter.measurements_string\n(\n timeseries_name String,\n timeseries_key UInt64,\n timestamp DateTime64(9, 'UTC'),\n datum Nullable(String)\n)\nENGINE = MergeTree()\nORDER BY (timeseries_name, timeseries_key, timestamp)\nTTL toDateTime(timestamp) + INTERVAL 30 DAY
30032025-04-30T06:42:46.225ZTRACtest_snapshot_basic (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 6ad06c6b-9a6e-4ed1-b3f4-5d2e9da3f8a7 sql = \n\nCREATE TABLE IF NOT EXISTS oximeter.measurements_bytes\n(\n timeseries_name String,\n timeseries_key UInt64,\n timestamp DateTime64(9, 'UTC'),\n /*\n * NOTE: Right now we can't unambiguously record a nullable byte array.\n * Arrays cannot be nested in `Nullable()` types, and encoding the array as\n * a string isn't palatable for a few reasons.\n * See: https://github.com/oxidecomputer/omicron/issues/4551 for more\n * details.\n */\n datum Array(UInt8)\n)\nENGINE = MergeTree()\nORDER BY (timeseries_name, timeseries_key, timestamp)\nTTL toDateTime(timestamp) + INTERVAL 30 DAY
30042025-04-30T06:42:46.228ZTRACtest_snapshot_basic (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 6ad06c6b-9a6e-4ed1-b3f4-5d2e9da3f8a7 sql = \n\nCREATE TABLE IF NOT EXISTS oximeter.measurements_cumulativei64\n(\n timeseries_name String,\n timeseries_key UInt64,\n start_time DateTime64(9, 'UTC'),\n timestamp DateTime64(9, 'UTC'),\n datum Nullable(Int64)\n)\nENGINE = MergeTree()\nORDER BY (timeseries_name, timeseries_key, start_time, timestamp)\nTTL toDateTime(timestamp) + INTERVAL 30 DAY
30052025-04-30T06:42:46.232ZTRACtest_snapshot_basic (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 6ad06c6b-9a6e-4ed1-b3f4-5d2e9da3f8a7 sql = \n\nCREATE TABLE IF NOT EXISTS oximeter.measurements_cumulativeu64\n(\n timeseries_name String,\n timeseries_key UInt64,\n start_time DateTime64(9, 'UTC'),\n timestamp DateTime64(9, 'UTC'),\n datum Nullable(UInt64)\n)\nENGINE = MergeTree()\nORDER BY (timeseries_name, timeseries_key, start_time, timestamp)\nTTL toDateTime(timestamp) + INTERVAL 30 DAY
30062025-04-30T06:42:46.240ZTRACtest_snapshot_basic (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 6ad06c6b-9a6e-4ed1-b3f4-5d2e9da3f8a7 sql = \n\nCREATE TABLE IF NOT EXISTS oximeter.measurements_cumulativef32\n(\n timeseries_name String,\n timeseries_key UInt64,\n start_time DateTime64(9, 'UTC'),\n timestamp DateTime64(9, 'UTC'),\n datum Nullable(Float32)\n)\nENGINE = MergeTree()\nORDER BY (timeseries_name, timeseries_key, start_time, timestamp)\nTTL toDateTime(timestamp) + INTERVAL 30 DAY
30072025-04-30T06:42:46.247ZTRACtest_snapshot_basic (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 6ad06c6b-9a6e-4ed1-b3f4-5d2e9da3f8a7 sql = \n\n\nCREATE TABLE IF NOT EXISTS oximeter.measurements_cumulativef64\n(\n timeseries_name String,\n timeseries_key UInt64,\n start_time DateTime64(9, 'UTC'),\n timestamp DateTime64(9, 'UTC'),\n datum Nullable(Float64)\n)\nENGINE = MergeTree()\nORDER BY (timeseries_name, timeseries_key, start_time, timestamp)\nTTL toDateTime(timestamp) + INTERVAL 30 DAY
30082025-04-30T06:42:46.251ZTRACtest_snapshot_basic (dns): buffer buffer = Length: 79 (0x4f) bytes\n0000: 51 5a 01 00 00 01 00 00 00 00 00 01 10 5f 63 72 QZ..........._cr\n0010: 75 63 69 62 6c 65 2d 70 61 6e 74 72 79 04 5f 74 ucible-pantry._t\n0020: 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 cp.control-plane\n0030: 05 6f 78 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 .oxide.internal.\n0040: 00 21 00 01 00 00 29 04 d0 00 00 00 00 00 00 .!....)........ kind = dns peer_addr = [::1]:51999 req_id = bef874bd-16ad-4ce1-9bcc-c65dfedaadc7
30092025-04-30T06:42:46.251ZDEBGtest_snapshot_basic (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 20826,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_crucible-pantry._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_crucible-pantry._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 16,\n 95,\n 99,\n 114,\n 117,\n 99,\n 105,\n 98,\n 108,\n 101,\n 45,\n 112,\n 97,\n 110,\n 116,\n 114,\n 121,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:51999 req_id = bef874bd-16ad-4ce1-9bcc-c65dfedaadc7
30102025-04-30T06:42:46.251ZDEBGtest_snapshot_basic (store): query key key = _crucible-pantry._tcp
30112025-04-30T06:42:46.252ZDEBGtest_snapshot_basic (store): query key key = 92ab293e-632a-4f12-897f-edcc61e0fc3b.host
30122025-04-30T06:42:46.252ZDEBGtest_snapshot_basic (dns): dns response additional_records = [Record { name_labels: Name("92ab293e-632a-4f12-897f-edcc61e0fc3b.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:51999 query = LowerQuery { name: LowerName(Name("_crucible-pantry._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_crucible-pantry._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_crucible-pantry._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 62733, target: Name("92ab293e-632a-4f12-897f-edcc61e0fc3b.host.control-plane.oxide.internal") })) }] req_id = bef874bd-16ad-4ce1-9bcc-c65dfedaadc7
30132025-04-30T06:42:46.252ZTRACtest_snapshot_basic (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 6ad06c6b-9a6e-4ed1-b3f4-5d2e9da3f8a7 sql = \n\nCREATE TABLE IF NOT EXISTS oximeter.measurements_histogrami8\n(\n timeseries_name String,\n timeseries_key UInt64,\n start_time DateTime64(9, 'UTC'),\n timestamp DateTime64(9, 'UTC'),\n /*\n * NOTE: Array types cannot be Nullable, see\n * https://clickhouse.com/docs/en/sql-reference/data-types/nullable\n * for more details.\n *\n * This means we need to use empty arrays to indicate a missing value. This\n * is unfortunate, and at this point relies on the fact that an\n * `oximeter::Histogram` cannot have zero bins. If that changes, we'll need\n * to figure out another way to represent missing samples here.\n */\n bins Array(Int8),\n counts Array(UInt64),\n min Int8,\n max Int8,\n sum_of_samples Int64,\n squared_mean Float64,\n p50_marker_heights Array(Float64),\n p50_marker_positions Array(UInt64),\n p50_desired_marker_positions Array(Float64),\n p90_marker_heights Array(Float64),\n p90_marker_positions Array(UInt64),\n p90_desired_marker_positions Array(Float64),\n p99_marker_heights Array(Float64),\n p99_marker_positions Array(UInt64),\n p99_desired_marker_positions Array(Float64)\n)\nENGINE = MergeTree()\nORDER BY (timeseries_name, timeseries_key, start_time, timestamp)\nTTL toDateTime(timestamp) + INTERVAL 30 DAY
30142025-04-30T06:42:46.253ZTRACtest_snapshot_basic (dns): buffer buffer = Length: 99 (0x63) bytes\n0000: 9f b0 01 00 00 01 00 00 00 00 00 01 24 39 32 61 ............$92a\n0010: 62 32 39 33 65 2d 36 33 32 61 2d 34 66 31 32 2d b293e-632a-4f12-\n0020: 38 39 37 66 2d 65 64 63 63 36 31 65 30 66 63 33 897f-edcc61e0fc3\n0030: 62 04 68 6f 73 74 0d 63 6f 6e 74 72 6f 6c 2d 70 b.host.control-p\n0040: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0050: 6e 61 6c 00 00 1c 00 01 00 00 29 04 d0 00 00 00 nal.......).....\n0060: 00 00 00 ... kind = dns peer_addr = [::1]:60215 req_id = d3e18498-879e-43a0-97cc-e96ad03af5fe
30152025-04-30T06:42:46.253ZDEBGtest_snapshot_basic (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 40880,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("92ab293e-632a-4f12-897f-edcc61e0fc3b.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("92ab293e-632a-4f12-897f-edcc61e0fc3b.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 36,\n 57,\n 50,\n 97,\n 98,\n 50,\n 57,\n 51,\n 101,\n 45,\n 54,\n 51,\n 50,\n 97,\n 45,\n 52,\n 102,\n 49,\n 50,\n 45,\n 56,\n 57,\n 55,\n 102,\n 45,\n 101,\n 100,\n 99,\n 99,\n 54,\n 49,\n 101,\n 48,\n 102,\n 99,\n 51,\n 98,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:60215 req_id = d3e18498-879e-43a0-97cc-e96ad03af5fe
30162025-04-30T06:42:46.254ZDEBGtest_snapshot_basic (store): query key key = 92ab293e-632a-4f12-897f-edcc61e0fc3b.host
30172025-04-30T06:42:46.254ZDEBGtest_snapshot_basic (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:60215 query = LowerQuery { name: LowerName(Name("92ab293e-632a-4f12-897f-edcc61e0fc3b.host.control-plane.oxide.internal.")), original: Query { name: Name("92ab293e-632a-4f12-897f-edcc61e0fc3b.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("92ab293e-632a-4f12-897f-edcc61e0fc3b.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = d3e18498-879e-43a0-97cc-e96ad03af5fe
30182025-04-30T06:42:46.321ZTRACtest_snapshot_basic (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 6ad06c6b-9a6e-4ed1-b3f4-5d2e9da3f8a7 sql = \n\nCREATE TABLE IF NOT EXISTS oximeter.measurements_histogramu8\n(\n timeseries_name String,\n timeseries_key UInt64,\n start_time DateTime64(9, 'UTC'),\n timestamp DateTime64(9, 'UTC'),\n bins Array(UInt8),\n counts Array(UInt64),\n min UInt8,\n max UInt8,\n sum_of_samples Int64,\n squared_mean Float64,\n p50_marker_heights Array(Float64),\n p50_marker_positions Array(UInt64),\n p50_desired_marker_positions Array(Float64),\n p90_marker_heights Array(Float64),\n p90_marker_positions Array(UInt64),\n p90_desired_marker_positions Array(Float64),\n p99_marker_heights Array(Float64),\n p99_marker_positions Array(UInt64),\n p99_desired_marker_positions Array(Float64)\n)\nENGINE = MergeTree()\nORDER BY (timeseries_name, timeseries_key, start_time, timestamp)\nTTL toDateTime(timestamp) + INTERVAL 30 DAY
30192025-04-30T06:42:46.323ZINFOtest_snapshot_basic (dropshot): accepted connection kind = pantry local_addr = [::1]:62733 remote_addr = [::1]:36381 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
30202025-04-30T06:42:46.325ZTRACtest_snapshot_basic (dropshot): incoming request kind = pantry local_addr = [::1]:62733 method = GET remote_addr = [::1]:36381 req_id = 12a51d2f-fa6a-44e9-9bd7-f56dd6cda758 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /crucible/pantry/0
30212025-04-30T06:42:46.326ZINFOtest_snapshot_basic (dropshot): request completed kind = pantry latency_us = 597 local_addr = [::1]:62733 method = GET remote_addr = [::1]:36381 req_id = 12a51d2f-fa6a-44e9-9bd7-f56dd6cda758 response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /crucible/pantry/0
30222025-04-30T06:42:46.851ZTRACtest_snapshot_basic (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 6ad06c6b-9a6e-4ed1-b3f4-5d2e9da3f8a7 sql = \n\nCREATE TABLE IF NOT EXISTS oximeter.measurements_histogrami16\n(\n timeseries_name String,\n timeseries_key UInt64,\n start_time DateTime64(9, 'UTC'),\n timestamp DateTime64(9, 'UTC'),\n bins Array(Int16),\n counts Array(UInt64),\n min Int16,\n max Int16,\n sum_of_samples Int64,\n squared_mean Float64,\n p50_marker_heights Array(Float64),\n p50_marker_positions Array(UInt64),\n p50_desired_marker_positions Array(Float64),\n p90_marker_heights Array(Float64),\n p90_marker_positions Array(UInt64),\n p90_desired_marker_positions Array(Float64),\n p99_marker_heights Array(Float64),\n p99_marker_positions Array(UInt64),\n p99_desired_marker_positions Array(Float64)\n)\nENGINE = MergeTree()\nORDER BY (timeseries_name, timeseries_key, start_time, timestamp)\nTTL toDateTime(timestamp) + INTERVAL 30 DAY
30232025-04-30T06:42:46.852ZINFOtest_snapshot_basic (dropshot): accepted connection kind = pantry local_addr = [::1]:62733 remote_addr = [::1]:44814 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
30242025-04-30T06:42:46.852ZINFOtest_snapshot_basic (dropshot): accepted connection kind = pantry local_addr = [::1]:62733 remote_addr = [::1]:60833 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
30252025-04-30T06:42:46.852ZINFOtest_snapshot_basic (dropshot): accepted connection kind = pantry local_addr = [::1]:62733 remote_addr = [::1]:32821 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
30262025-04-30T06:42:46.852ZINFOtest_snapshot_basic (dropshot): accepted connection kind = pantry local_addr = [::1]:62733 remote_addr = [::1]:44150 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
30272025-04-30T06:42:46.852ZINFOtest_snapshot_basic (dropshot): accepted connection kind = pantry local_addr = [::1]:62733 remote_addr = [::1]:51509 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
30282025-04-30T06:42:46.852ZINFOtest_snapshot_basic (dropshot): accepted connection kind = pantry local_addr = [::1]:62733 remote_addr = [::1]:60712 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
30292025-04-30T06:42:46.853ZINFOtest_snapshot_basic (dropshot): accepted connection kind = pantry local_addr = [::1]:62733 remote_addr = [::1]:45987 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
30302025-04-30T06:42:46.854ZDEBGtest_snapshot_basic (producer-registration-task): will register with Nexus at 127.0.0.1:61468
30312025-04-30T06:42:46.921ZDEBGtest_snapshot_basic (producer-registration-task): client request body = Some(Body) method = POST uri = http://127.0.0.1:61468/metrics/producers
30322025-04-30T06:42:46.925ZTRACtest_snapshot_basic (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 6ad06c6b-9a6e-4ed1-b3f4-5d2e9da3f8a7 sql = \n\nCREATE TABLE IF NOT EXISTS oximeter.measurements_histogramu16\n(\n timeseries_name String,\n timeseries_key UInt64,\n start_time DateTime64(9, 'UTC'),\n timestamp DateTime64(9, 'UTC'),\n bins Array(UInt16),\n counts Array(UInt64),\n min UInt16,\n max UInt16,\n sum_of_samples Int64,\n squared_mean Float64,\n p50_marker_heights Array(Float64),\n p50_marker_positions Array(UInt64),\n p50_desired_marker_positions Array(Float64),\n p90_marker_heights Array(Float64),\n p90_marker_positions Array(UInt64),\n p90_desired_marker_positions Array(Float64),\n p99_marker_heights Array(Float64),\n p99_marker_positions Array(UInt64),\n p99_desired_marker_positions Array(Float64)\n)\nENGINE = MergeTree()\nORDER BY (timeseries_name, timeseries_key, start_time, timestamp)\nTTL toDateTime(timestamp) + INTERVAL 30 DAY
30332025-04-30T06:42:46.926ZINFOtest_snapshot_basic (dropshot_internal): accepted connection local_addr = 127.0.0.1:61468 remote_addr = 127.0.0.1:38673
30342025-04-30T06:42:46.926ZTRACtest_snapshot_basic (dropshot): incoming request kind = pantry local_addr = [::1]:62733 method = GET remote_addr = [::1]:44814 req_id = 19db3161-030c-4063-8dd8-54b3296d484d sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /crucible/pantry/0
30352025-04-30T06:42:46.926ZTRACtest_snapshot_basic (dropshot): incoming request kind = pantry local_addr = [::1]:62733 method = GET remote_addr = [::1]:60833 req_id = 15db898f-cec4-4d38-bad9-631e584562b9 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /crucible/pantry/0
30362025-04-30T06:42:46.926ZTRACtest_snapshot_basic (dropshot): incoming request kind = pantry local_addr = [::1]:62733 method = GET remote_addr = [::1]:32821 req_id = 93b5ab0c-7e6a-46da-a550-2e728672a203 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /crucible/pantry/0
30372025-04-30T06:42:46.927ZTRACtest_snapshot_basic (dropshot): incoming request kind = pantry local_addr = [::1]:62733 method = GET remote_addr = [::1]:44150 req_id = 1801bfd5-6010-4272-8069-0b85d2ad3171 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /crucible/pantry/0
30382025-04-30T06:42:46.929ZTRACtest_snapshot_basic (dropshot): incoming request kind = pantry local_addr = [::1]:62733 method = GET remote_addr = [::1]:51509 req_id = 8db9fc31-6ab0-4f36-a93c-e324c8463477 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /crucible/pantry/0
30392025-04-30T06:42:46.929ZTRACtest_snapshot_basic (dropshot): incoming request kind = pantry local_addr = [::1]:62733 method = GET remote_addr = [::1]:60712 req_id = 54f1847e-d5c5-4fcc-98f6-27f64ab89e6c sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /crucible/pantry/0
30402025-04-30T06:42:46.929ZTRACtest_snapshot_basic (dropshot): incoming request kind = pantry local_addr = [::1]:62733 method = GET remote_addr = [::1]:45987 req_id = 4a521d61-5528-4557-a2a9-e538a6c9c97b sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /crucible/pantry/0
30412025-04-30T06:42:46.930ZINFOtest_snapshot_basic (dropshot): request completed kind = pantry latency_us = 3596 local_addr = [::1]:62733 method = GET remote_addr = [::1]:44814 req_id = 19db3161-030c-4063-8dd8-54b3296d484d response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /crucible/pantry/0
30422025-04-30T06:42:46.930ZINFOtest_snapshot_basic (dropshot): request completed kind = pantry latency_us = 3524 local_addr = [::1]:62733 method = GET remote_addr = [::1]:60833 req_id = 15db898f-cec4-4d38-bad9-631e584562b9 response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /crucible/pantry/0
30432025-04-30T06:42:46.930ZINFOtest_snapshot_basic (dropshot): request completed kind = pantry latency_us = 3446 local_addr = [::1]:62733 method = GET remote_addr = [::1]:32821 req_id = 93b5ab0c-7e6a-46da-a550-2e728672a203 response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /crucible/pantry/0
30442025-04-30T06:42:46.930ZINFOtest_snapshot_basic (dropshot): request completed kind = pantry latency_us = 3222 local_addr = [::1]:62733 method = GET remote_addr = [::1]:44150 req_id = 1801bfd5-6010-4272-8069-0b85d2ad3171 response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /crucible/pantry/0
30452025-04-30T06:42:46.930ZINFOtest_snapshot_basic (dropshot): request completed kind = pantry latency_us = 1038 local_addr = [::1]:62733 method = GET remote_addr = [::1]:51509 req_id = 8db9fc31-6ab0-4f36-a93c-e324c8463477 response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /crucible/pantry/0
30462025-04-30T06:42:46.930ZINFOtest_snapshot_basic (dropshot): request completed kind = pantry latency_us = 943 local_addr = [::1]:62733 method = GET remote_addr = [::1]:60712 req_id = 54f1847e-d5c5-4fcc-98f6-27f64ab89e6c response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /crucible/pantry/0
30472025-04-30T06:42:46.930ZINFOtest_snapshot_basic (dropshot): request completed kind = pantry latency_us = 852 local_addr = [::1]:62733 method = GET remote_addr = [::1]:45987 req_id = 4a521d61-5528-4557-a2a9-e538a6c9c97b response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /crucible/pantry/0
30482025-04-30T06:42:46.930ZTRACtest_snapshot_basic (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 6ad06c6b-9a6e-4ed1-b3f4-5d2e9da3f8a7 sql = \n\nCREATE TABLE IF NOT EXISTS oximeter.measurements_histogrami32\n(\n timeseries_name String,\n timeseries_key UInt64,\n start_time DateTime64(9, 'UTC'),\n timestamp DateTime64(9, 'UTC'),\n bins Array(Int32),\n counts Array(UInt64),\n min Int32,\n max Int32,\n sum_of_samples Int64,\n squared_mean Float64,\n p50_marker_heights Array(Float64),\n p50_marker_positions Array(UInt64),\n p50_desired_marker_positions Array(Float64),\n p90_marker_heights Array(Float64),\n p90_marker_positions Array(UInt64),\n p90_desired_marker_positions Array(Float64),\n p99_marker_heights Array(Float64),\n p99_marker_positions Array(UInt64),\n p99_desired_marker_positions Array(Float64)\n)\nENGINE = MergeTree()\nORDER BY (timeseries_name, timeseries_key, start_time, timestamp)\nTTL toDateTime(timestamp) + INTERVAL 30 DAY
30492025-04-30T06:42:46.932ZTRACtest_snapshot_basic (dropshot_internal): incoming request local_addr = 127.0.0.1:61468 method = POST remote_addr = 127.0.0.1:38673 req_id = 24abbb19-8346-4066-a562-1d8d035a8434 uri = /metrics/producers
30502025-04-30T06:42:46.935ZTRACtest_snapshot_basic (dropshot_internal): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:61468 method = POST remote_addr = 127.0.0.1:38673 req_id = 24abbb19-8346-4066-a562-1d8d035a8434 resource = Database uri = /metrics/producers
30512025-04-30T06:42:46.935ZDEBGtest_snapshot_basic (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:61468 method = POST remote_addr = 127.0.0.1:38673 req_id = 24abbb19-8346-4066-a562-1d8d035a8434 roles = RoleSet { roles: {} } uri = /metrics/producers
30522025-04-30T06:42:46.935ZDEBGtest_snapshot_basic (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:61468 method = POST remote_addr = 127.0.0.1:38673 req_id = 24abbb19-8346-4066-a562-1d8d035a8434 resource = Database result = Ok(()) uri = /metrics/producers
30532025-04-30T06:42:46.937ZTRACtest_snapshot_basic (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 6ad06c6b-9a6e-4ed1-b3f4-5d2e9da3f8a7 sql = \n\nCREATE TABLE IF NOT EXISTS oximeter.measurements_histogramu32\n(\n timeseries_name String,\n timeseries_key UInt64,\n start_time DateTime64(9, 'UTC'),\n timestamp DateTime64(9, 'UTC'),\n bins Array(UInt32),\n counts Array(UInt64),\n min UInt32,\n max UInt32,\n sum_of_samples Int64,\n squared_mean Float64,\n p50_marker_heights Array(Float64),\n p50_marker_positions Array(UInt64),\n p50_desired_marker_positions Array(Float64),\n p90_marker_heights Array(Float64),\n p90_marker_positions Array(UInt64),\n p90_desired_marker_positions Array(Float64),\n p99_marker_heights Array(Float64),\n p99_marker_positions Array(UInt64),\n p99_desired_marker_positions Array(Float64)\n)\nENGINE = MergeTree()\nORDER BY (timeseries_name, timeseries_key, start_time, timestamp)\nTTL toDateTime(timestamp) + INTERVAL 30 DAY
30542025-04-30T06:42:46.945ZTRACtest_snapshot_basic (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 6ad06c6b-9a6e-4ed1-b3f4-5d2e9da3f8a7 sql = \n\nCREATE TABLE IF NOT EXISTS oximeter.measurements_histogrami64\n(\n timeseries_name String,\n timeseries_key UInt64,\n start_time DateTime64(9, 'UTC'),\n timestamp DateTime64(9, 'UTC'),\n bins Array(Int64),\n counts Array(UInt64),\n min Int64,\n max Int64,\n sum_of_samples Int64,\n squared_mean Float64,\n p50_marker_heights Array(Float64),\n p50_marker_positions Array(UInt64),\n p50_desired_marker_positions Array(Float64),\n p90_marker_heights Array(Float64),\n p90_marker_positions Array(UInt64),\n p90_desired_marker_positions Array(Float64),\n p99_marker_heights Array(Float64),\n p99_marker_positions Array(UInt64),\n p99_desired_marker_positions Array(Float64)\n)\nENGINE = MergeTree()\nORDER BY (timeseries_name, timeseries_key, start_time, timestamp)\nTTL toDateTime(timestamp) + INTERVAL 30 DAY
30552025-04-30T06:42:46.949ZINFOtest_snapshot_basic (dropshot_internal): request completed error_message_external = Service Unavailable error_message_internal = no Oximeter instances available for assignment latency_us = 16983 local_addr = 127.0.0.1:61468 method = POST remote_addr = 127.0.0.1:38673 req_id = 24abbb19-8346-4066-a562-1d8d035a8434 response_code = 503 uri = /metrics/producers
30562025-04-30T06:42:46.949ZWARNtest_snapshot_basic (datastore): Non-retryable transaction failure err = DatabaseError(UniqueViolation, "duplicate key value violates unique constraint \\"inv_caboose_pkey\\"") transaction = inventory_insert_collection
30572025-04-30T06:42:46.949ZWARNtest_snapshot_basic (BackgroundTasks): inventory collection failed background_task = inventory_collection error = failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint "inv_caboose_pkey"
30582025-04-30T06:42:46.949ZDEBGtest_snapshot_basic (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 2.047937441s iteration = 3 status = {"error":"failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint \\"inv_caboose_pkey\\""}
30592025-04-30T06:42:46.950ZTRACtest_snapshot_basic (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 6ad06c6b-9a6e-4ed1-b3f4-5d2e9da3f8a7 sql = \n\nCREATE TABLE IF NOT EXISTS oximeter.measurements_histogramu64\n(\n timeseries_name String,\n timeseries_key UInt64,\n start_time DateTime64(9, 'UTC'),\n timestamp DateTime64(9, 'UTC'),\n bins Array(UInt64),\n counts Array(UInt64),\n min UInt64,\n max UInt64,\n sum_of_samples Int64,\n squared_mean Float64,\n p50_marker_heights Array(Float64),\n p50_marker_positions Array(UInt64),\n p50_desired_marker_positions Array(Float64),\n p90_marker_heights Array(Float64),\n p90_marker_positions Array(UInt64),\n p90_desired_marker_positions Array(Float64),\n p99_marker_heights Array(Float64),\n p99_marker_positions Array(UInt64),\n p99_desired_marker_positions Array(Float64)\n)\nENGINE = MergeTree()\nORDER BY (timeseries_name, timeseries_key, start_time, timestamp)\nTTL toDateTime(timestamp) + INTERVAL 30 DAY
30602025-04-30T06:42:46.957ZDEBGtest_snapshot_basic (producer-registration-task): client response result = Ok(Response { url: "http://127.0.0.1:61468/metrics/producers", status: 503, headers: {"content-type": "application/json", "x-request-id": "24abbb19-8346-4066-a562-1d8d035a8434", "content-length": "133", "date": "Wed, 30 Apr 2025 06:42:46 GMT"} })
30612025-04-30T06:42:46.959ZWARNtest_snapshot_basic (producer-registration-task): failed to register with Nexus, will retry count = 1 delay = 1.019470929s error = "Error Response: status: 503 Service Unavailable; headers: {\\"content-type\\": \\"application/json\\", \\"x-request-id\\": \\"24abbb19-8346-4066-a562-1d8d035a8434\\", \\"content-length\\": \\"133\\", \\"date\\": \\"Wed, 30 Apr 2025 06:42:46 GMT\\"}; value: Error { error_code: Some(\\"ServiceNotAvailable\\"), message: \\"Service Unavailable\\", request_id: \\"24abbb19-8346-4066-a562-1d8d035a8434\\" }" file = oximeter/producer/src/lib.rs:371
30622025-04-30T06:42:46.959ZTRACtest_snapshot_basic (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 6ad06c6b-9a6e-4ed1-b3f4-5d2e9da3f8a7 sql = \n\nCREATE TABLE IF NOT EXISTS oximeter.measurements_histogramf32\n(\n timeseries_name String,\n timeseries_key UInt64,\n start_time DateTime64(9, 'UTC'),\n timestamp DateTime64(9, 'UTC'),\n bins Array(Float32),\n counts Array(UInt64),\n min Float32,\n max Float32,\n sum_of_samples Float64,\n squared_mean Float64,\n p50_marker_heights Array(Float64),\n p50_marker_positions Array(UInt64),\n p50_desired_marker_positions Array(Float64),\n p90_marker_heights Array(Float64),\n p90_marker_positions Array(UInt64),\n p90_desired_marker_positions Array(Float64),\n p99_marker_heights Array(Float64),\n p99_marker_positions Array(UInt64),\n p99_desired_marker_positions Array(Float64)\n)\nENGINE = MergeTree()\nORDER BY (timeseries_name, timeseries_key, start_time, timestamp)\nTTL toDateTime(timestamp) + INTERVAL 30 DAY
30632025-04-30T06:42:46.964ZTRACtest_snapshot_basic (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 6ad06c6b-9a6e-4ed1-b3f4-5d2e9da3f8a7 sql = \n\nCREATE TABLE IF NOT EXISTS oximeter.measurements_histogramf64\n(\n timeseries_name String,\n timeseries_key UInt64,\n start_time DateTime64(9, 'UTC'),\n timestamp DateTime64(9, 'UTC'),\n bins Array(Float64),\n counts Array(UInt64),\n min Float64,\n max Float64,\n sum_of_samples Float64,\n squared_mean Float64,\n p50_marker_heights Array(Float64),\n p50_marker_positions Array(UInt64),\n p50_desired_marker_positions Array(Float64),\n p90_marker_heights Array(Float64),\n p90_marker_positions Array(UInt64),\n p90_desired_marker_positions Array(Float64),\n p99_marker_heights Array(Float64),\n p99_marker_positions Array(UInt64),\n p99_desired_marker_positions Array(Float64)\n)\nENGINE = MergeTree()\nORDER BY (timeseries_name, timeseries_key, start_time, timestamp)\nTTL toDateTime(timestamp) + INTERVAL 30 DAY
30642025-04-30T06:42:46.967ZTRACtest_snapshot_basic (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 6ad06c6b-9a6e-4ed1-b3f4-5d2e9da3f8a7 sql = \n\n/* The field tables store named dimensions of each timeseries.\n *\n * As with the measurement tables, there is one field table for each field data\n * type. Fields are deduplicated by using the "replacing merge tree", though\n * this behavior **must not** be relied upon for query correctness.\n *\n * The index for the fields differs from the measurements, however. Rows are\n * sorted by timeseries name, then field name, field value, and finally\n * timeseries key. This reflects the most common pattern for looking them up:\n * by field name and possibly value, within a timeseries. The resulting keys are\n * usually then used to look up measurements.\n *\n * NOTE: We may want to consider a secondary index on these tables, sorting by\n * timeseries name and then key, since it would improve lookups where one\n * already has the key. Realistically though, these tables are quite small and\n * so performance benefits will be low in absolute terms.\n *\n * TTL: We use a materialized column to expire old field table records. This\n * column is generated automatically by the database whenever a new row is\n * inserted. It cannot be inserted directly, nor is it returned in a `SELECT *`\n * query. Since these tables are `ReplacingMergeTree`s, that means the last\n * record will remain during a deduplication, which will have the last\n * timestamp. ClickHouse will then expire old data for us, similar to the\n * measurement tables.\n */\nCREATE TABLE IF NOT EXISTS oximeter.fields_bool\n(\n timeseries_name String,\n timeseries_key UInt64,\n field_name String,\n field_value Bool,\n last_updated_at DateTime MATERIALIZED now()\n)\nENGINE = ReplacingMergeTree()\nORDER BY (timeseries_name, field_name, field_value, timeseries_key)\nTTL last_updated_at + INTERVAL 30 DAY
30652025-04-30T06:42:46.971ZTRACtest_snapshot_basic (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 6ad06c6b-9a6e-4ed1-b3f4-5d2e9da3f8a7 sql = \n\nCREATE TABLE IF NOT EXISTS oximeter.fields_i8\n(\n timeseries_name String,\n timeseries_key UInt64,\n field_name String,\n field_value Int8,\n last_updated_at DateTime MATERIALIZED now()\n)\nENGINE = ReplacingMergeTree()\nORDER BY (timeseries_name, field_name, field_value, timeseries_key)\nTTL last_updated_at + INTERVAL 30 DAY
30662025-04-30T06:42:46.974ZTRACtest_snapshot_basic (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 6ad06c6b-9a6e-4ed1-b3f4-5d2e9da3f8a7 sql = \n\nCREATE TABLE IF NOT EXISTS oximeter.fields_u8\n(\n timeseries_name String,\n timeseries_key UInt64,\n field_name String,\n field_value UInt8,\n last_updated_at DateTime MATERIALIZED now()\n)\nENGINE = ReplacingMergeTree()\nORDER BY (timeseries_name, field_name, field_value, timeseries_key)\nTTL last_updated_at + INTERVAL 30 DAY
30672025-04-30T06:42:46.982ZTRACtest_snapshot_basic (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 6ad06c6b-9a6e-4ed1-b3f4-5d2e9da3f8a7 sql = \n\nCREATE TABLE IF NOT EXISTS oximeter.fields_i16\n(\n timeseries_name String,\n timeseries_key UInt64,\n field_name String,\n field_value Int16,\n last_updated_at DateTime MATERIALIZED now()\n)\nENGINE = ReplacingMergeTree()\nORDER BY (timeseries_name, field_name, field_value, timeseries_key)\nTTL last_updated_at + INTERVAL 30 DAY
30682025-04-30T06:42:46.986ZTRACtest_snapshot_basic (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 6ad06c6b-9a6e-4ed1-b3f4-5d2e9da3f8a7 sql = \n\nCREATE TABLE IF NOT EXISTS oximeter.fields_u16\n(\n timeseries_name String,\n timeseries_key UInt64,\n field_name String,\n field_value UInt16,\n last_updated_at DateTime MATERIALIZED now()\n)\nENGINE = ReplacingMergeTree()\nORDER BY (timeseries_name, field_name, field_value, timeseries_key)\nTTL last_updated_at + INTERVAL 30 DAY
30692025-04-30T06:42:46.989ZTRACtest_snapshot_basic (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 6ad06c6b-9a6e-4ed1-b3f4-5d2e9da3f8a7 sql = \n\nCREATE TABLE IF NOT EXISTS oximeter.fields_i32\n(\n timeseries_name String,\n timeseries_key UInt64,\n field_name String,\n field_value Int32,\n last_updated_at DateTime MATERIALIZED now()\n)\nENGINE = ReplacingMergeTree()\nORDER BY (timeseries_name, field_name, field_value, timeseries_key)\nTTL last_updated_at + INTERVAL 30 DAY
30702025-04-30T06:42:46.993ZTRACtest_snapshot_basic (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 6ad06c6b-9a6e-4ed1-b3f4-5d2e9da3f8a7 sql = \n\nCREATE TABLE IF NOT EXISTS oximeter.fields_u32\n(\n timeseries_name String,\n timeseries_key UInt64,\n field_name String,\n field_value UInt32,\n last_updated_at DateTime MATERIALIZED now()\n)\nENGINE = ReplacingMergeTree()\nORDER BY (timeseries_name, field_name, field_value, timeseries_key)\nTTL last_updated_at + INTERVAL 30 DAY
30712025-04-30T06:42:46.998ZTRACtest_snapshot_basic (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 6ad06c6b-9a6e-4ed1-b3f4-5d2e9da3f8a7 sql = \n\nCREATE TABLE IF NOT EXISTS oximeter.fields_i64\n(\n timeseries_name String,\n timeseries_key UInt64,\n field_name String,\n field_value Int64,\n last_updated_at DateTime MATERIALIZED now()\n)\nENGINE = ReplacingMergeTree()\nORDER BY (timeseries_name, field_name, field_value, timeseries_key)\nTTL last_updated_at + INTERVAL 30 DAY
30722025-04-30T06:42:47.006ZTRACtest_snapshot_basic (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 6ad06c6b-9a6e-4ed1-b3f4-5d2e9da3f8a7 sql = \n\nCREATE TABLE IF NOT EXISTS oximeter.fields_u64\n(\n timeseries_name String,\n timeseries_key UInt64,\n field_name String,\n field_value UInt64,\n last_updated_at DateTime MATERIALIZED now()\n)\nENGINE = ReplacingMergeTree()\nORDER BY (timeseries_name, field_name, field_value, timeseries_key)\nTTL last_updated_at + INTERVAL 30 DAY
30732025-04-30T06:42:47.010ZTRACtest_snapshot_basic (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 6ad06c6b-9a6e-4ed1-b3f4-5d2e9da3f8a7 sql = \n\nCREATE TABLE IF NOT EXISTS oximeter.fields_ipaddr\n(\n timeseries_name String,\n timeseries_key UInt64,\n field_name String,\n field_value IPv6,\n last_updated_at DateTime MATERIALIZED now()\n)\nENGINE = ReplacingMergeTree()\nORDER BY (timeseries_name, field_name, field_value, timeseries_key)\nTTL last_updated_at + INTERVAL 30 DAY
30742025-04-30T06:42:47.013ZTRACtest_snapshot_basic (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 6ad06c6b-9a6e-4ed1-b3f4-5d2e9da3f8a7 sql = \n\nCREATE TABLE IF NOT EXISTS oximeter.fields_string\n(\n timeseries_name String,\n timeseries_key UInt64,\n field_name String,\n field_value String,\n last_updated_at DateTime MATERIALIZED now()\n)\nENGINE = ReplacingMergeTree()\nORDER BY (timeseries_name, field_name, field_value, timeseries_key)\nTTL last_updated_at + INTERVAL 30 DAY
30752025-04-30T06:42:47.015ZTRACtest_snapshot_basic (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 6ad06c6b-9a6e-4ed1-b3f4-5d2e9da3f8a7 sql = \n\nCREATE TABLE IF NOT EXISTS oximeter.fields_uuid\n(\n timeseries_name String,\n timeseries_key UInt64,\n field_name String,\n field_value UUID,\n last_updated_at DateTime MATERIALIZED now()\n)\nENGINE = ReplacingMergeTree()\nORDER BY (timeseries_name, field_name, field_value, timeseries_key)\nTTL last_updated_at + INTERVAL 30 DAY
30762025-04-30T06:42:47.018ZTRACtest_snapshot_basic (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 6ad06c6b-9a6e-4ed1-b3f4-5d2e9da3f8a7 sql = \n\n/* The timeseries schema table stores the extracted schema for the samples\n * oximeter collects.\n */\nCREATE TABLE IF NOT EXISTS oximeter.timeseries_schema\n(\n timeseries_name String,\n fields Nested(\n name String,\n type Enum(\n 'Bool' = 1,\n 'I64' = 2,\n 'IpAddr' = 3,\n 'String' = 4,\n 'Uuid' = 6,\n 'I8' = 7,\n 'U8' = 8,\n 'I16' = 9,\n 'U16' = 10,\n 'I32' = 11,\n 'U32' = 12,\n 'U64' = 13\n ),\n source Enum(\n 'Target' = 1,\n 'Metric' = 2\n )\n ),\n datum_type Enum(\n 'Bool' = 1,\n 'I64' = 2,\n 'F64' = 3,\n 'String' = 4,\n 'Bytes' = 5,\n 'CumulativeI64' = 6,\n 'CumulativeF64' = 7,\n 'HistogramI64' = 8,\n 'HistogramF64' = 9,\n 'I8' = 10,\n 'U8' = 11,\n 'I16' = 12,\n 'U16' = 13,\n 'I32' = 14,\n 'U32' = 15,\n 'U64' = 16,\n 'F32' = 17,\n 'CumulativeU64' = 18,\n 'CumulativeF32' = 19,\n 'HistogramI8' = 20,\n 'HistogramU8' = 21,\n 'HistogramI16' = 22,\n 'HistogramU16' = 23,\n 'HistogramI32' = 24,\n 'HistogramU32' = 25,\n 'HistogramU64' = 26,\n 'HistogramF32' = 27\n ),\n created DateTime64(9, 'UTC')\n)\nENGINE = MergeTree()\nORDER BY (timeseries_name, fields.name)
30772025-04-30T06:42:47.022ZINFOtest_snapshot_basic (clickhouse-client): inserting current version collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 file = oximeter/db/src/client/mod.rs:825 id = 6ad06c6b-9a6e-4ed1-b3f4-5d2e9da3f8a7 version = 14
30782025-04-30T06:42:47.022ZTRACtest_snapshot_basic (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 6ad06c6b-9a6e-4ed1-b3f4-5d2e9da3f8a7 sql = INSERT INTO oximeter.version (*) VALUES (14, now());
30792025-04-30T06:42:47.028ZDEBGtest_snapshot_basic (clickhouse-client): registered USDT probes collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 4b38df0f-201b-4c5c-aa13-a20876ffc923
30802025-04-30T06:42:47.031ZDEBGtest_snapshot_basic (dropshot): registered endpoint local_addr = [::1]:54951 method = GET path = /info versions = all
30812025-04-30T06:42:47.031ZDEBGtest_snapshot_basic (dropshot): registered endpoint local_addr = [::1]:54951 method = GET path = /producers versions = all
30822025-04-30T06:42:47.031ZDEBGtest_snapshot_basic (dropshot): registered endpoint local_addr = [::1]:54951 method = DELETE path = /producers/{producer_id} versions = all
30832025-04-30T06:42:47.031ZDEBGtest_snapshot_basic (dropshot): registered endpoint local_addr = [::1]:54951 method = GET path = /producers/{producer_id} versions = all
30842025-04-30T06:42:47.031ZINFOtest_snapshot_basic (dropshot): listening file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:328 local_addr = [::1]:54951
30852025-04-30T06:42:47.033ZDEBGtest_snapshot_basic (dropshot): successfully registered DTrace USDT probes local_addr = [::1]:54951
30862025-04-30T06:42:47.034ZDEBGtest_snapshot_basic (oximeter): registered USDT probes
30872025-04-30T06:42:47.034ZDEBGtest_snapshot_basic (oximeter): contacting nexus
30882025-04-30T06:42:47.593ZDEBGtest_snapshot_basic (oximeter): client request body = Some(Body) method = POST uri = http://127.0.0.1:61468/metrics/collectors
30892025-04-30T06:42:47.593ZDEBGtest_snapshot_basic (producer-registration-task): will register with Nexus at 127.0.0.1:61468
30902025-04-30T06:42:47.669ZDEBGtest_snapshot_basic (producer-registration-task): client request body = Some(Body) method = POST uri = http://127.0.0.1:61468/metrics/producers
30912025-04-30T06:42:47.677ZINFOtest_snapshot_basic (dropshot_internal): accepted connection local_addr = 127.0.0.1:61468 remote_addr = 127.0.0.1:64332
30922025-04-30T06:42:47.677ZINFOtest_snapshot_basic (dropshot_internal): accepted connection local_addr = 127.0.0.1:61468 remote_addr = 127.0.0.1:35781
30932025-04-30T06:42:47.678ZTRACtest_snapshot_basic (dropshot_internal): incoming request local_addr = 127.0.0.1:61468 method = POST remote_addr = 127.0.0.1:35781 req_id = 07a2d0df-956e-4830-afac-09ee8b513036 uri = /metrics/producers
30942025-04-30T06:42:47.680ZTRACtest_snapshot_basic (dropshot_internal): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:61468 method = POST remote_addr = 127.0.0.1:35781 req_id = 07a2d0df-956e-4830-afac-09ee8b513036 resource = Database uri = /metrics/producers
30952025-04-30T06:42:47.680ZDEBGtest_snapshot_basic (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:61468 method = POST remote_addr = 127.0.0.1:35781 req_id = 07a2d0df-956e-4830-afac-09ee8b513036 roles = RoleSet { roles: {} } uri = /metrics/producers
30962025-04-30T06:42:47.692ZDEBGtest_snapshot_basic (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:61468 method = POST remote_addr = 127.0.0.1:35781 req_id = 07a2d0df-956e-4830-afac-09ee8b513036 resource = Database result = Ok(()) uri = /metrics/producers
30972025-04-30T06:42:47.692ZTRACtest_snapshot_basic (dropshot_internal): incoming request local_addr = 127.0.0.1:61468 method = POST remote_addr = 127.0.0.1:64332 req_id = 796ae571-f060-457d-a19c-b11a32b8590e uri = /metrics/collectors
30982025-04-30T06:42:47.692ZTRACtest_snapshot_basic (dropshot_internal): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:61468 method = POST remote_addr = 127.0.0.1:64332 req_id = 796ae571-f060-457d-a19c-b11a32b8590e resource = Database uri = /metrics/collectors
30992025-04-30T06:42:47.693ZDEBGtest_snapshot_basic (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:61468 method = POST remote_addr = 127.0.0.1:64332 req_id = 796ae571-f060-457d-a19c-b11a32b8590e roles = RoleSet { roles: {} } uri = /metrics/collectors
31002025-04-30T06:42:47.693ZDEBGtest_snapshot_basic (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:61468 method = POST remote_addr = 127.0.0.1:64332 req_id = 796ae571-f060-457d-a19c-b11a32b8590e resource = Database result = Ok(()) uri = /metrics/collectors
31012025-04-30T06:42:47.702ZINFOtest_snapshot_basic (dropshot_internal): request completed error_message_external = Service Unavailable error_message_internal = no Oximeter instances available for assignment latency_us = 23677 local_addr = 127.0.0.1:61468 method = POST remote_addr = 127.0.0.1:35781 req_id = 07a2d0df-956e-4830-afac-09ee8b513036 response_code = 503 uri = /metrics/producers
31022025-04-30T06:42:47.704ZDEBGtest_snapshot_basic (producer-registration-task): client response result = Ok(Response { url: "http://127.0.0.1:61468/metrics/producers", status: 503, headers: {"content-type": "application/json", "x-request-id": "07a2d0df-956e-4830-afac-09ee8b513036", "content-length": "133", "date": "Wed, 30 Apr 2025 06:42:47 GMT"} })
31032025-04-30T06:42:47.707ZWARNtest_snapshot_basic (producer-registration-task): failed to register with Nexus, will retry count = 2 delay = 1.766954712s error = "Error Response: status: 503 Service Unavailable; headers: {\\"content-type\\": \\"application/json\\", \\"x-request-id\\": \\"07a2d0df-956e-4830-afac-09ee8b513036\\", \\"content-length\\": \\"133\\", \\"date\\": \\"Wed, 30 Apr 2025 06:42:47 GMT\\"}; value: Error { error_code: Some(\\"ServiceNotAvailable\\"), message: \\"Service Unavailable\\", request_id: \\"07a2d0df-956e-4830-afac-09ee8b513036\\" }" file = oximeter/producer/src/lib.rs:371
31042025-04-30T06:42:47.707ZINFOtest_snapshot_basic (nexus): registered new oximeter metric collection server address = [::1]:54951 collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78
31052025-04-30T06:42:47.707ZINFOtest_snapshot_basic (dropshot_internal): request completed latency_us = 15991 local_addr = 127.0.0.1:61468 method = POST remote_addr = 127.0.0.1:64332 req_id = 796ae571-f060-457d-a19c-b11a32b8590e response_code = 204 uri = /metrics/collectors
31062025-04-30T06:42:47.713ZDEBGtest_snapshot_basic (oximeter): client response result = Ok(Response { url: "http://127.0.0.1:61468/metrics/collectors", status: 204, headers: {"x-request-id": "796ae571-f060-457d-a19c-b11a32b8590e", "date": "Wed, 30 Apr 2025 06:42:47 GMT"} })
31072025-04-30T06:42:47.713ZINFOtest_snapshot_basic (oximeter): oximeter registered with nexus file = oximeter/collector/src/lib.rs:392 id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78
31082025-04-30T06:42:47.713ZDEBGtest_snapshot_basic (init_with_steps): Running step start_producer_server
31092025-04-30T06:42:47.713ZDEBGtest_snapshot_basic: Starting test metric Producer Server
31102025-04-30T06:42:47.717ZDEBGtest_snapshot_basic (Simulated CrucibleAgent Dropshot Server): registered endpoint kind = storage local_addr = [::1]:59241 method = GET path = /crucible/0/regions server = b6d65341-167c-41df-9b5c-41cded99c229 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
31112025-04-30T06:42:47.717ZDEBGtest_snapshot_basic (Simulated CrucibleAgent Dropshot Server): registered endpoint kind = storage local_addr = [::1]:59241 method = POST path = /crucible/0/regions server = b6d65341-167c-41df-9b5c-41cded99c229 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
31122025-04-30T06:42:47.717ZDEBGtest_snapshot_basic (Simulated CrucibleAgent Dropshot Server): registered endpoint kind = storage local_addr = [::1]:59241 method = DELETE path = /crucible/0/regions/{id} server = b6d65341-167c-41df-9b5c-41cded99c229 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
31132025-04-30T06:42:47.717ZDEBGtest_snapshot_basic (Simulated CrucibleAgent Dropshot Server): registered endpoint kind = storage local_addr = [::1]:59241 method = GET path = /crucible/0/regions/{id} server = b6d65341-167c-41df-9b5c-41cded99c229 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
31142025-04-30T06:42:47.717ZDEBGtest_snapshot_basic (Simulated CrucibleAgent Dropshot Server): registered endpoint kind = storage local_addr = [::1]:59241 method = GET path = /crucible/0/regions/{id}/snapshots server = b6d65341-167c-41df-9b5c-41cded99c229 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
31152025-04-30T06:42:47.717ZDEBGtest_snapshot_basic (Simulated CrucibleAgent Dropshot Server): registered endpoint kind = storage local_addr = [::1]:59241 method = DELETE path = /crucible/0/regions/{id}/snapshots/{name} server = b6d65341-167c-41df-9b5c-41cded99c229 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
31162025-04-30T06:42:47.717ZDEBGtest_snapshot_basic (Simulated CrucibleAgent Dropshot Server): registered endpoint kind = storage local_addr = [::1]:59241 method = GET path = /crucible/0/regions/{id}/snapshots/{name} server = b6d65341-167c-41df-9b5c-41cded99c229 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
31172025-04-30T06:42:47.717ZDEBGtest_snapshot_basic (Simulated CrucibleAgent Dropshot Server): registered endpoint kind = storage local_addr = [::1]:59241 method = DELETE path = /crucible/0/regions/{id}/snapshots/{name}/run server = b6d65341-167c-41df-9b5c-41cded99c229 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
31182025-04-30T06:42:47.717ZDEBGtest_snapshot_basic (Simulated CrucibleAgent Dropshot Server): registered endpoint kind = storage local_addr = [::1]:59241 method = POST path = /crucible/0/regions/{id}/snapshots/{name}/run server = b6d65341-167c-41df-9b5c-41cded99c229 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
31192025-04-30T06:42:47.717ZINFOtest_snapshot_basic (Simulated CrucibleAgent Dropshot Server): listening kind = storage local_addr = [::1]:59241 server = b6d65341-167c-41df-9b5c-41cded99c229 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
31202025-04-30T06:42:47.717ZDEBGtest_snapshot_basic (Simulated CrucibleAgent Dropshot Server): successfully registered DTrace USDT probes kind = storage local_addr = [::1]:59241 server = b6d65341-167c-41df-9b5c-41cded99c229 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
31212025-04-30T06:42:47.717ZINFOtest_snapshot_basic (SledAgent): Created Simulated Crucible Server address = [::1]:59241 kind = storage server = b6d65341-167c-41df-9b5c-41cded99c229 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
31222025-04-30T06:42:47.718ZINFOtest_snapshot_basic (nexus): inserting test physical disk model = test-model physical_disk_id = 4a18174e-2451-4ec5-a7ec-2ab2d6caf55b serial = totally-unique-serial: 4a18174e-2451-4ec5-a7ec-2ab2d6caf55b sled_id = b6d65341-167c-41df-9b5c-41cded99c229 vendor = test-vendor
31232025-04-30T06:42:47.718ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
31242025-04-30T06:42:47.718ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {} }
31252025-04-30T06:42:47.718ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
31262025-04-30T06:42:47.720ZINFOtest_snapshot_basic (oximeter-agent): starting refresh list task collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 file = oximeter/collector/src/agent.rs:559
31272025-04-30T06:42:47.799ZINFOtest_snapshot_basic (dropshot_internal): accepted connection local_addr = 127.0.0.1:61468 remote_addr = 127.0.0.1:51491
31282025-04-30T06:42:47.800ZINFOtest_snapshot_basic (oximeter-agent): refreshing list of producers from Nexus collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 file = oximeter/collector/src/agent.rs:562
31292025-04-30T06:42:47.805ZDEBGtest_snapshot_basic (oximeter): client request body = None method = GET uri = http://127.0.0.1:61468/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
31302025-04-30T06:42:47.805ZTRACtest_snapshot_basic (dropshot_internal): incoming request local_addr = 127.0.0.1:61468 method = GET remote_addr = 127.0.0.1:64332 req_id = 2dc33092-0d60-4e4b-8232-7d5529c03e01 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
31312025-04-30T06:42:47.806ZTRACtest_snapshot_basic (dropshot_internal): incoming request local_addr = 127.0.0.1:61468 method = POST remote_addr = 127.0.0.1:51491 req_id = ecadf5cc-b2b1-4579-a742-ea470f35b867 uri = /metrics/producers
31322025-04-30T06:42:47.808ZTRACtest_snapshot_basic (dropshot_internal): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:61468 method = GET remote_addr = 127.0.0.1:64332 req_id = 2dc33092-0d60-4e4b-8232-7d5529c03e01 resource = Database uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
31332025-04-30T06:42:47.808ZDEBGtest_snapshot_basic (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:61468 method = GET remote_addr = 127.0.0.1:64332 req_id = 2dc33092-0d60-4e4b-8232-7d5529c03e01 roles = RoleSet { roles: {} } uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
31342025-04-30T06:42:47.809ZDEBGtest_snapshot_basic (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:61468 method = GET remote_addr = 127.0.0.1:64332 req_id = 2dc33092-0d60-4e4b-8232-7d5529c03e01 resource = Database result = Ok(()) uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
31352025-04-30T06:42:47.810ZTRACtest_snapshot_basic (dropshot_internal): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:61468 method = POST remote_addr = 127.0.0.1:51491 req_id = ecadf5cc-b2b1-4579-a742-ea470f35b867 resource = Database uri = /metrics/producers
31362025-04-30T06:42:47.810ZDEBGtest_snapshot_basic (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:61468 method = POST remote_addr = 127.0.0.1:51491 req_id = ecadf5cc-b2b1-4579-a742-ea470f35b867 roles = RoleSet { roles: {} } uri = /metrics/producers
31372025-04-30T06:42:47.810ZDEBGtest_snapshot_basic (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:61468 method = POST remote_addr = 127.0.0.1:51491 req_id = ecadf5cc-b2b1-4579-a742-ea470f35b867 resource = Database result = Ok(()) uri = /metrics/producers
31382025-04-30T06:42:47.811ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
31392025-04-30T06:42:47.811ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {} }
31402025-04-30T06:42:47.811ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
31412025-04-30T06:42:47.816ZINFOtest_snapshot_basic (dropshot_internal): request completed latency_us = 10879 local_addr = 127.0.0.1:61468 method = GET remote_addr = 127.0.0.1:64332 req_id = 2dc33092-0d60-4e4b-8232-7d5529c03e01 response_code = 200 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
31422025-04-30T06:42:47.821ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Fleet
31432025-04-30T06:42:47.821ZTRACtest_snapshot_basic (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
31442025-04-30T06:42:47.821ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
31452025-04-30T06:42:47.821ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {} }
31462025-04-30T06:42:47.822ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
31472025-04-30T06:42:47.823ZDEBGtest_snapshot_basic (oximeter): client response result = Ok(Response { url: "http://127.0.0.1:61468/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending", status: 200, headers: {"content-type": "application/json", "x-request-id": "2dc33092-0d60-4e4b-8232-7d5529c03e01", "content-length": "29", "date": "Wed, 30 Apr 2025 06:42:47 GMT"} })
31482025-04-30T06:42:47.823ZINFOtest_snapshot_basic (oximeter-agent): refreshed list of producers from Nexus collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 file = oximeter/collector/src/agent.rs:637 n_current_tasks = 0 n_pruned_tasks = 0
31492025-04-30T06:42:47.836ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
31502025-04-30T06:42:47.852ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Fleet result = Ok(())
31512025-04-30T06:42:47.871ZINFOtest_snapshot_basic (nexus): assigned collector to new producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
31522025-04-30T06:42:47.871ZINFOtest_snapshot_basic (dropshot_internal): request completed latency_us = 65654 local_addr = 127.0.0.1:61468 method = POST remote_addr = 127.0.0.1:51491 req_id = ecadf5cc-b2b1-4579-a742-ea470f35b867 response_code = 201 uri = /metrics/producers
31532025-04-30T06:42:47.977ZINFOtest_snapshot_basic (nexus): upserting zpool physical_disk_id = 4a18174e-2451-4ec5-a7ec-2ab2d6caf55b sled_id = b6d65341-167c-41df-9b5c-41cded99c229 zpool_id = f5a7d0c8-2111-4f7a-8212-3c5695b6dceb
31542025-04-30T06:42:47.977ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
31552025-04-30T06:42:47.977ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {} }
31562025-04-30T06:42:47.978ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
31572025-04-30T06:42:47.988ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Fleet
31582025-04-30T06:42:47.988ZTRACtest_snapshot_basic (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
31592025-04-30T06:42:47.988ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
31602025-04-30T06:42:47.988ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {} }
31612025-04-30T06:42:47.990ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
31622025-04-30T06:42:47.994ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
31632025-04-30T06:42:47.996ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Fleet result = Ok(())
31642025-04-30T06:42:48.006ZINFOtest_snapshot_basic (nexus): upserting Crucible dataset dataset_id = ecbc0ed8-b081-4d94-a6b7-a3374503b36c zpool_id = f5a7d0c8-2111-4f7a-8212-3c5695b6dceb
31652025-04-30T06:42:48.015ZINFOtest_snapshot_basic (nexus): inserting test physical disk model = test-model physical_disk_id = 4a18174e-2451-4ec5-a7ec-2ab2d6caf55b serial = totally-unique-serial: 4a18174e-2451-4ec5-a7ec-2ab2d6caf55b sled_id = b6d65341-167c-41df-9b5c-41cded99c229 vendor = test-vendor
31662025-04-30T06:42:48.018ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
31672025-04-30T06:42:48.018ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {} }
31682025-04-30T06:42:48.019ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
31692025-04-30T06:42:48.025ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = PhysicalDisk { parent: Fleet, key: 4a18174e-2451-4ec5-a7ec-2ab2d6caf55b (physical_disk), lookup_type: ById(4a18174e-2451-4ec5-a7ec-2ab2d6caf55b) }
31702025-04-30T06:42:48.025ZTRACtest_snapshot_basic (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
31712025-04-30T06:42:48.025ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
31722025-04-30T06:42:48.025ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {} }
31732025-04-30T06:42:48.026ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
31742025-04-30T06:42:48.029ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
31752025-04-30T06:42:48.035ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = PhysicalDisk { parent: Fleet, key: 4a18174e-2451-4ec5-a7ec-2ab2d6caf55b (physical_disk), lookup_type: ById(4a18174e-2451-4ec5-a7ec-2ab2d6caf55b) } result = Ok(())
31762025-04-30T06:42:48.035ZINFOtest_snapshot_basic (nexus): upserting zpool physical_disk_id = 4a18174e-2451-4ec5-a7ec-2ab2d6caf55b sled_id = b6d65341-167c-41df-9b5c-41cded99c229 zpool_id = f5a7d0c8-2111-4f7a-8212-3c5695b6dceb
31772025-04-30T06:42:48.035ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
31782025-04-30T06:42:48.035ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {} }
31792025-04-30T06:42:48.035ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
31802025-04-30T06:42:48.036ZTRACtest_snapshot_basic (results-sink): batch interval expired, but no samples to insert collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
31812025-04-30T06:42:48.036ZTRACtest_snapshot_basic (results-sink-cluster): batch interval expired, but no samples to insert collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
31822025-04-30T06:42:48.037ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Fleet
31832025-04-30T06:42:48.037ZTRACtest_snapshot_basic (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
31842025-04-30T06:42:48.037ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
31852025-04-30T06:42:48.037ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {} }
31862025-04-30T06:42:48.038ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
31872025-04-30T06:42:48.040ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
31882025-04-30T06:42:48.045ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Fleet result = Ok(())
31892025-04-30T06:42:48.058ZTRACtest_snapshot_basic (InternalApi): authorize begin action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Fleet
31902025-04-30T06:42:48.059ZTRACtest_snapshot_basic (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
31912025-04-30T06:42:48.059ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
31922025-04-30T06:42:48.059ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {} }
31932025-04-30T06:42:48.060ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
31942025-04-30T06:42:48.080ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
31952025-04-30T06:42:48.083ZDEBGtest_snapshot_basic (InternalApi): authorize result action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Fleet result = Ok(())
31962025-04-30T06:42:48.083ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
31972025-04-30T06:42:48.083ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {} }
31982025-04-30T06:42:48.084ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
31992025-04-30T06:42:48.090ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
32002025-04-30T06:42:48.090ZTRACtest_snapshot_basic (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
32012025-04-30T06:42:48.091ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
32022025-04-30T06:42:48.091ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {} }
32032025-04-30T06:42:48.091ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
32042025-04-30T06:42:48.091ZDEBGtest_snapshot_basic (BackgroundTasks): activating background_task = inventory_collection iteration = 4 reason = Signaled
32052025-04-30T06:42:48.091ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
32062025-04-30T06:42:48.092ZTRACtest_snapshot_basic (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
32072025-04-30T06:42:48.092ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
32082025-04-30T06:42:48.092ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
32092025-04-30T06:42:48.092ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
32102025-04-30T06:42:48.120ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
32112025-04-30T06:42:48.127ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
32122025-04-30T06:42:48.131ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
32132025-04-30T06:42:48.131ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {} }
32142025-04-30T06:42:48.131ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
32152025-04-30T06:42:48.136ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
32162025-04-30T06:42:48.142ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
32172025-04-30T06:42:48.143ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
32182025-04-30T06:42:48.143ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
32192025-04-30T06:42:48.143ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
32202025-04-30T06:42:48.150ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
32212025-04-30T06:42:48.150ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {} }
32222025-04-30T06:42:48.150ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
32232025-04-30T06:42:48.151ZDEBGtest_snapshot_basic (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = [(d1ad44e5-18dd-4193-a70e-bf9042f0e0e1, 0)]
32242025-04-30T06:42:48.151ZTRACtest_snapshot_basic (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
32252025-04-30T06:42:48.151ZTRACtest_snapshot_basic (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: 18 48 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 .H..........._mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:56791 req_id = a62f4784-8057-4894-ad8a-a1dbc68cfc66
32262025-04-30T06:42:48.151ZDEBGtest_snapshot_basic (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 6216,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:56791 req_id = a62f4784-8057-4894-ad8a-a1dbc68cfc66
32272025-04-30T06:42:48.151ZDEBGtest_snapshot_basic (store): query key key = _mgs._tcp
32282025-04-30T06:42:48.152ZDEBGtest_snapshot_basic (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
32292025-04-30T06:42:48.154ZDEBGtest_snapshot_basic (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:56791 query = LowerQuery { name: LowerName(Name("_mgs._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 41502, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = a62f4784-8057-4894-ad8a-a1dbc68cfc66
32302025-04-30T06:42:48.155ZDEBGtest_snapshot_basic (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 41502, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 4232, tv_nsec: 954689527 } })
32312025-04-30T06:42:48.155ZTRACtest_snapshot_basic (DnsResolver): lookup_service_targets: looking up SRV target
32322025-04-30T06:42:48.156ZTRACtest_snapshot_basic (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: 4f 45 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e OE..........-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:62536 req_id = 9e022b5b-2cce-496f-bd9a-55127dc8fb0f
32332025-04-30T06:42:48.156ZDEBGtest_snapshot_basic (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 20293,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:62536 req_id = 9e022b5b-2cce-496f-bd9a-55127dc8fb0f
32342025-04-30T06:42:48.157ZDEBGtest_snapshot_basic (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
32352025-04-30T06:42:48.157ZDEBGtest_snapshot_basic (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:62536 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = 9e022b5b-2cce-496f-bd9a-55127dc8fb0f
32362025-04-30T06:42:48.229ZTRACtest_snapshot_basic (DnsResolver): lookup_all_socket_v6 srv dns_name = _clickhouse-admin-keeper._tcp.control-plane.oxide.internal
32372025-04-30T06:42:48.230ZTRACtest_snapshot_basic (dns): buffer buffer = Length: 87 (0x57) bytes\n0000: 23 e4 01 00 00 01 00 00 00 00 00 01 18 5f 63 6c #............_cl\n0010: 69 63 6b 68 6f 75 73 65 2d 61 64 6d 69 6e 2d 6b ickhouse-admin-k\n0020: 65 65 70 65 72 04 5f 74 63 70 0d 63 6f 6e 74 72 eeper._tcp.contr\n0030: 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 69 ol-plane.oxide.i\n0040: 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 04 nternal..!....).\n0050: d0 00 00 00 00 00 00 ....... kind = dns peer_addr = [::1]:57977 req_id = 981f89d2-4d6f-4bcf-b7d6-4628df476ac9
32382025-04-30T06:42:48.230ZDEBGtest_snapshot_basic (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 9188,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 24,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 97,\n 100,\n 109,\n 105,\n 110,\n 45,\n 107,\n 101,\n 101,\n 112,\n 101,\n 114,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:57977 req_id = 981f89d2-4d6f-4bcf-b7d6-4628df476ac9
32392025-04-30T06:42:48.230ZDEBGtest_snapshot_basic (store): query key key = _clickhouse-admin-keeper._tcp
32402025-04-30T06:42:48.230ZERROtest_snapshot_basic (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 9188, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } NXDOMAIN: no records found for name: "_clickhouse-admin-keeper._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:57977 req_id = 981f89d2-4d6f-4bcf-b7d6-4628df476ac9
32412025-04-30T06:42:48.232ZDEBGtest_snapshot_basic (BackgroundTasks): begin collection background_task = inventory_collection
32422025-04-30T06:42:48.232ZDEBGtest_snapshot_basic (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:41502
32432025-04-30T06:42:48.232ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/ignition
32442025-04-30T06:42:48.246ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "29e04ffb-f2c3-42b5-9800-e193806d3e49", "content-length": "835", "date": "Wed, 30 Apr 2025 06:42:47 GMT"} })
32452025-04-30T06:42:48.246ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/0
32462025-04-30T06:42:48.248ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "d5bbf32c-1da5-47fb-85c2-10cf9062bedb", "content-length": "734", "date": "Wed, 30 Apr 2025 06:42:47 GMT"} })
32472025-04-30T06:42:48.249ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/0/component/sp/caboose?firmware_slot=0
32482025-04-30T06:42:48.254ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "efc61348-a507-4483-8e63-92333fba6547", "content-length": "111", "date": "Wed, 30 Apr 2025 06:42:47 GMT"} })
32492025-04-30T06:42:48.254ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/0/component/sp/caboose?firmware_slot=1
32502025-04-30T06:42:48.254ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "9ed858b3-104e-4272-86a9-127b832d6a73", "content-length": "111", "date": "Wed, 30 Apr 2025 06:42:47 GMT"} })
32512025-04-30T06:42:48.255ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/0/component/rot/caboose?firmware_slot=0
32522025-04-30T06:42:48.257ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "ec071f07-b19b-4185-b517-a67981648d57", "content-length": "170", "date": "Wed, 30 Apr 2025 06:42:47 GMT"} })
32532025-04-30T06:42:48.257ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/0/component/rot/caboose?firmware_slot=1
32542025-04-30T06:42:48.259ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "7cbe5e09-c861-4186-baa5-52623dc3b134", "content-length": "170", "date": "Wed, 30 Apr 2025 06:42:47 GMT"} })
32552025-04-30T06:42:48.259ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/0/component/stage0/caboose?firmware_slot=0
32562025-04-30T06:42:48.262ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "a76c31d6-5393-4f3f-a60c-d264d01de137", "content-length": "179", "date": "Wed, 30 Apr 2025 06:42:47 GMT"} })
32572025-04-30T06:42:48.262ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/0/component/stage0/caboose?firmware_slot=1
32582025-04-30T06:42:48.266ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "13881c20-54cc-41d1-95a0-9c50d74235f9", "content-length": "179", "date": "Wed, 30 Apr 2025 06:42:47 GMT"} })
32592025-04-30T06:42:48.266ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/0/component/rot/cmpa
32602025-04-30T06:42:48.267ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "bb90beef-a346-4cc3-a6dc-8c88058879d8", "content-length": "702", "date": "Wed, 30 Apr 2025 06:42:47 GMT"} })
32612025-04-30T06:42:48.267ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/0/component/rot/cfpa
32622025-04-30T06:42:48.269ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "7fb0c8b2-4674-42d2-af72-3c68ec4b0e54", "content-length": "727", "date": "Wed, 30 Apr 2025 06:42:47 GMT"} })
32632025-04-30T06:42:48.269ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/0/component/rot/cfpa
32642025-04-30T06:42:48.270ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "fb3c2c8d-4a25-4113-b2f7-7fafc645941c", "content-length": "729", "date": "Wed, 30 Apr 2025 06:42:47 GMT"} })
32652025-04-30T06:42:48.271ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/0/component/rot/cfpa
32662025-04-30T06:42:48.273ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "c9010c3b-bb61-48db-98cc-613050d6e035", "content-length": "728", "date": "Wed, 30 Apr 2025 06:42:47 GMT"} })
32672025-04-30T06:42:48.273ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/1
32682025-04-30T06:42:48.275ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "bf737991-e905-4c7b-839b-1ab21b461f66", "content-length": "734", "date": "Wed, 30 Apr 2025 06:42:47 GMT"} })
32692025-04-30T06:42:48.275ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/1/component/sp/caboose?firmware_slot=0
32702025-04-30T06:42:48.277ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "2fd0fba6-5ff8-45ec-8c1c-a194adc53eb7", "content-length": "111", "date": "Wed, 30 Apr 2025 06:42:47 GMT"} })
32712025-04-30T06:42:48.277ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/1/component/sp/caboose?firmware_slot=1
32722025-04-30T06:42:48.280ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "83a0f685-908b-41e2-a26b-e802f4846499", "content-length": "111", "date": "Wed, 30 Apr 2025 06:42:47 GMT"} })
32732025-04-30T06:42:48.280ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/1/component/rot/caboose?firmware_slot=0
32742025-04-30T06:42:48.282ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "85f399bc-7e47-4ba2-ae7d-5f8c7ddf38a3", "content-length": "170", "date": "Wed, 30 Apr 2025 06:42:47 GMT"} })
32752025-04-30T06:42:48.282ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/1/component/rot/caboose?firmware_slot=1
32762025-04-30T06:42:48.284ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "fcd8eb17-986b-4c7b-a3b5-854506601430", "content-length": "170", "date": "Wed, 30 Apr 2025 06:42:47 GMT"} })
32772025-04-30T06:42:48.284ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/1/component/stage0/caboose?firmware_slot=0
32782025-04-30T06:42:48.286ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "9a6c17b5-2ce6-4155-807a-82960b34da31", "content-length": "179", "date": "Wed, 30 Apr 2025 06:42:47 GMT"} })
32792025-04-30T06:42:48.286ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/1/component/stage0/caboose?firmware_slot=1
32802025-04-30T06:42:48.289ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "b2951c2e-95a0-41e1-8453-fc9fafec88af", "content-length": "179", "date": "Wed, 30 Apr 2025 06:42:47 GMT"} })
32812025-04-30T06:42:48.290ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/1/component/rot/cmpa
32822025-04-30T06:42:48.294ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "b963fbe6-ae76-473f-a9ed-5a6364ced5ca", "content-length": "702", "date": "Wed, 30 Apr 2025 06:42:47 GMT"} })
32832025-04-30T06:42:48.294ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/1/component/rot/cfpa
32842025-04-30T06:42:48.296ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "e923852d-5b2d-4cb6-9aa2-e533328766bd", "content-length": "727", "date": "Wed, 30 Apr 2025 06:42:47 GMT"} })
32852025-04-30T06:42:48.296ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/1/component/rot/cfpa
32862025-04-30T06:42:48.296ZDEBGtest_snapshot_basic: attempt to fetch latest inventory collection result = Ok("found")
32872025-04-30T06:42:48.298ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "6882e458-bf27-40cd-97a2-35977e3d4a76", "content-length": "729", "date": "Wed, 30 Apr 2025 06:42:47 GMT"} })
32882025-04-30T06:42:48.298ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/1/component/rot/cfpa
32892025-04-30T06:42:48.299ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "5b30f7c6-4cc9-40aa-810d-a283dc53990b", "content-length": "728", "date": "Wed, 30 Apr 2025 06:42:47 GMT"} })
32902025-04-30T06:42:48.300ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/0
32912025-04-30T06:42:48.302ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "16bd769a-683e-4f66-b300-e4ef013e9bc8", "content-length": "723", "date": "Wed, 30 Apr 2025 06:42:47 GMT"} })
32922025-04-30T06:42:48.302ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/0/component/sp/caboose?firmware_slot=0
32932025-04-30T06:42:48.305ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "441d79e1-20c8-41f8-a45c-958fd1371a0c", "content-length": "109", "date": "Wed, 30 Apr 2025 06:42:47 GMT"} })
32942025-04-30T06:42:48.305ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/0/component/sp/caboose?firmware_slot=1
32952025-04-30T06:42:48.312ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "89187e71-9ac6-4c74-8708-16be42c64ab5", "content-length": "109", "date": "Wed, 30 Apr 2025 06:42:47 GMT"} })
32962025-04-30T06:42:48.312ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/0/component/rot/caboose?firmware_slot=0
32972025-04-30T06:42:48.316ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "444b9657-8b9a-476a-9986-3063fcf4c20b", "content-length": "169", "date": "Wed, 30 Apr 2025 06:42:47 GMT"} })
32982025-04-30T06:42:48.316ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/0/component/rot/caboose?firmware_slot=1
32992025-04-30T06:42:48.319ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "5d2e9fd6-5210-4e9f-ac9f-374aec4e5276", "content-length": "169", "date": "Wed, 30 Apr 2025 06:42:47 GMT"} })
33002025-04-30T06:42:48.319ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/0/component/stage0/caboose?firmware_slot=0
33012025-04-30T06:42:48.321ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "4f532dd2-51cd-4408-a6e7-e829cc8d37f3", "content-length": "178", "date": "Wed, 30 Apr 2025 06:42:47 GMT"} })
33022025-04-30T06:42:48.322ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/0/component/stage0/caboose?firmware_slot=1
33032025-04-30T06:42:48.330ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "95b11b85-30c1-452f-a860-d6a4e51a8d36", "content-length": "178", "date": "Wed, 30 Apr 2025 06:42:47 GMT"} })
33042025-04-30T06:42:48.331ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/0/component/rot/cmpa
33052025-04-30T06:42:48.331ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "ee4cb353-17aa-4fd5-ad57-6fadf772e2db", "content-length": "702", "date": "Wed, 30 Apr 2025 06:42:47 GMT"} })
33062025-04-30T06:42:48.332ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/0/component/rot/cfpa
33072025-04-30T06:42:48.333ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "5cb002e8-f374-4936-81c8-fda3f6976410", "content-length": "727", "date": "Wed, 30 Apr 2025 06:42:47 GMT"} })
33082025-04-30T06:42:48.333ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/0/component/rot/cfpa
33092025-04-30T06:42:48.339ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "ff2225b1-5efb-4569-a906-f331c7e4b862", "content-length": "729", "date": "Wed, 30 Apr 2025 06:42:47 GMT"} })
33102025-04-30T06:42:48.339ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/0/component/rot/cfpa
33112025-04-30T06:42:48.341ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "0ea583fe-7a7c-4f7f-a6c8-761e2330e365", "content-length": "728", "date": "Wed, 30 Apr 2025 06:42:47 GMT"} })
33122025-04-30T06:42:48.341ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/1
33132025-04-30T06:42:48.342ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "818d1eec-9a9b-46dd-bef4-6df15193b8f3", "content-length": "723", "date": "Wed, 30 Apr 2025 06:42:47 GMT"} })
33142025-04-30T06:42:48.343ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/1/component/sp/caboose?firmware_slot=0
33152025-04-30T06:42:48.346ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "6602f167-eb02-4cf0-bfda-ffeb3933dfb7", "content-length": "109", "date": "Wed, 30 Apr 2025 06:42:47 GMT"} })
33162025-04-30T06:42:48.347ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/1/component/sp/caboose?firmware_slot=1
33172025-04-30T06:42:48.348ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
33182025-04-30T06:42:48.348ZTRACtest_snapshot_basic (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
33192025-04-30T06:42:48.348ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
33202025-04-30T06:42:48.348ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {} }
33212025-04-30T06:42:48.349ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
33222025-04-30T06:42:48.352ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
33232025-04-30T06:42:48.359ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
33242025-04-30T06:42:48.359ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
33252025-04-30T06:42:48.359ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {} }
33262025-04-30T06:42:48.360ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
33272025-04-30T06:42:48.360ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "af1fca06-d84a-46a8-94b1-f0f944b278bf", "content-length": "109", "date": "Wed, 30 Apr 2025 06:42:47 GMT"} })
33282025-04-30T06:42:48.360ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/1/component/rot/caboose?firmware_slot=0
33292025-04-30T06:42:48.363ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
33302025-04-30T06:42:48.363ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {} }
33312025-04-30T06:42:48.364ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
33322025-04-30T06:42:48.368ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "0da0693e-385a-4743-a7db-6bb000dac7d7", "content-length": "169", "date": "Wed, 30 Apr 2025 06:42:47 GMT"} })
33332025-04-30T06:42:48.368ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/1/component/rot/caboose?firmware_slot=1
33342025-04-30T06:42:48.371ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "9f2c6f53-4776-4b6d-9889-9ad053fb08b6", "content-length": "169", "date": "Wed, 30 Apr 2025 06:42:47 GMT"} })
33352025-04-30T06:42:48.371ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/1/component/stage0/caboose?firmware_slot=0
33362025-04-30T06:42:48.374ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "afdd992a-c81a-4159-922c-1ee47da1c2c0", "content-length": "178", "date": "Wed, 30 Apr 2025 06:42:47 GMT"} })
33372025-04-30T06:42:48.374ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/1/component/stage0/caboose?firmware_slot=1
33382025-04-30T06:42:48.378ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "287a5678-b4af-4e9d-98e8-0864bc86f46a", "content-length": "178", "date": "Wed, 30 Apr 2025 06:42:47 GMT"} })
33392025-04-30T06:42:48.378ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/1/component/rot/cmpa
33402025-04-30T06:42:48.379ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "b17253c3-9ea1-47d4-81f2-a1f760724430", "content-length": "702", "date": "Wed, 30 Apr 2025 06:42:47 GMT"} })
33412025-04-30T06:42:48.380ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/1/component/rot/cfpa
33422025-04-30T06:42:48.382ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "fb3bacdd-4055-4d85-aa40-b3f20becf449", "content-length": "727", "date": "Wed, 30 Apr 2025 06:42:47 GMT"} })
33432025-04-30T06:42:48.383ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/1/component/rot/cfpa
33442025-04-30T06:42:48.385ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "0631defb-6ac7-4cc6-be8b-fd58d967f400", "content-length": "729", "date": "Wed, 30 Apr 2025 06:42:47 GMT"} })
33452025-04-30T06:42:48.386ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/1/component/rot/cfpa
33462025-04-30T06:42:48.387ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "620b6c67-0452-4ab7-82ce-1000a853054a", "content-length": "728", "date": "Wed, 30 Apr 2025 06:42:47 GMT"} })
33472025-04-30T06:42:48.388ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
33482025-04-30T06:42:48.388ZTRACtest_snapshot_basic (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
33492025-04-30T06:42:48.388ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
33502025-04-30T06:42:48.388ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
33512025-04-30T06:42:48.389ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
33522025-04-30T06:42:48.391ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
33532025-04-30T06:42:48.397ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
33542025-04-30T06:42:48.397ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
33552025-04-30T06:42:48.397ZTRACtest_snapshot_basic (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
33562025-04-30T06:42:48.397ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
33572025-04-30T06:42:48.397ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
33582025-04-30T06:42:48.397ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
33592025-04-30T06:42:48.400ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
33602025-04-30T06:42:48.407ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
33612025-04-30T06:42:48.407ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
33622025-04-30T06:42:48.407ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
33632025-04-30T06:42:48.408ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
33642025-04-30T06:42:48.409ZDEBGtest_snapshot_basic (producer-registration-task): will register with Nexus at 127.0.0.1:61468
33652025-04-30T06:42:48.503ZDEBGtest_snapshot_basic (producer-registration-task): client request body = Some(Body) method = POST uri = http://127.0.0.1:61468/metrics/producers
33662025-04-30T06:42:48.509ZINFOtest_snapshot_basic (dropshot_internal): accepted connection local_addr = 127.0.0.1:61468 remote_addr = 127.0.0.1:59701
33672025-04-30T06:42:48.576ZDEBGtest_snapshot_basic (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:41923
33682025-04-30T06:42:48.576ZDEBGtest_snapshot_basic (BackgroundTasks): client request SledAgent = http://[::1]:41923 background_task = inventory_collection body = None method = GET uri = http://[::1]:41923/inventory
33692025-04-30T06:42:48.580ZDEBGtest_snapshot_basic: attempt to fetch latest inventory collection result = Ok("found")
33702025-04-30T06:42:48.580ZINFOtest_snapshot_basic (dropshot): accepted connection local_addr = [::1]:41923 remote_addr = [::1]:36664 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
33712025-04-30T06:42:48.580ZTRACtest_snapshot_basic (dropshot_internal): incoming request local_addr = 127.0.0.1:61468 method = POST remote_addr = 127.0.0.1:59701 req_id = 8ff231c0-85e3-46f9-98f2-98f554dbc8af uri = /metrics/producers
33722025-04-30T06:42:48.582ZTRACtest_snapshot_basic (dropshot_internal): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:61468 method = POST remote_addr = 127.0.0.1:59701 req_id = 8ff231c0-85e3-46f9-98f2-98f554dbc8af resource = Database uri = /metrics/producers
33732025-04-30T06:42:48.582ZDEBGtest_snapshot_basic (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:61468 method = POST remote_addr = 127.0.0.1:59701 req_id = 8ff231c0-85e3-46f9-98f2-98f554dbc8af roles = RoleSet { roles: {} } uri = /metrics/producers
33742025-04-30T06:42:48.582ZDEBGtest_snapshot_basic (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:61468 method = POST remote_addr = 127.0.0.1:59701 req_id = 8ff231c0-85e3-46f9-98f2-98f554dbc8af resource = Database result = Ok(()) uri = /metrics/producers
33752025-04-30T06:42:48.585ZTRACtest_snapshot_basic (dropshot): incoming request local_addr = [::1]:41923 method = GET remote_addr = [::1]:36664 req_id = 713e7434-ab25-4836-a011-71b5ddcc7baf sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
33762025-04-30T06:42:48.586ZINFOtest_snapshot_basic (dropshot): request completed latency_us = 542 local_addr = [::1]:41923 method = GET remote_addr = [::1]:36664 req_id = 713e7434-ab25-4836-a011-71b5ddcc7baf response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
33772025-04-30T06:42:48.597ZDEBGtest_snapshot_basic (BackgroundTasks): client response SledAgent = http://[::1]:41923 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:41923/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "713e7434-ab25-4836-a011-71b5ddcc7baf", "content-length": "3141", "date": "Wed, 30 Apr 2025 06:42:47 GMT"} })
33782025-04-30T06:42:48.639ZDEBGtest_snapshot_basic (BackgroundTasks): begin collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
33792025-04-30T06:42:48.639ZDEBGtest_snapshot_basic (BackgroundTasks): end collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
33802025-04-30T06:42:48.639ZDEBGtest_snapshot_basic (BackgroundTasks): finished collection background_task = inventory_collection
33812025-04-30T06:42:48.645ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
33822025-04-30T06:42:48.645ZTRACtest_snapshot_basic (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
33832025-04-30T06:42:48.645ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
33842025-04-30T06:42:48.646ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
33852025-04-30T06:42:48.647ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
33862025-04-30T06:42:48.648ZINFOtest_snapshot_basic (nexus): assigned collector to new producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 producer_id = 354a52d4-3078-4706-81ce-12c32f6bbeaa
33872025-04-30T06:42:48.648ZINFOtest_snapshot_basic (dropshot_internal): request completed latency_us = 67636 local_addr = 127.0.0.1:61468 method = POST remote_addr = 127.0.0.1:59701 req_id = 8ff231c0-85e3-46f9-98f2-98f554dbc8af response_code = 201 uri = /metrics/producers
33882025-04-30T06:42:48.648ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
33892025-04-30T06:42:48.649ZTRACtest_snapshot_basic (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
33902025-04-30T06:42:48.649ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
33912025-04-30T06:42:48.649ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {} }
33922025-04-30T06:42:48.649ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
33932025-04-30T06:42:48.658ZDEBGtest_snapshot_basic (producer-registration-task): client response result = Ok(Response { url: "http://127.0.0.1:61468/metrics/producers", status: 201, headers: {"content-type": "application/json", "x-request-id": "8ff231c0-85e3-46f9-98f2-98f554dbc8af", "content-length": "41", "date": "Wed, 30 Apr 2025 06:42:47 GMT"} })
33942025-04-30T06:42:48.662ZDEBGtest_snapshot_basic (producer-registration-task): successfully registered with Nexus, pausing until time to renew lease lease_duration = 600s wait_period = 150s
33952025-04-30T06:42:48.700ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
33962025-04-30T06:42:48.707ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
33972025-04-30T06:42:48.707ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
33982025-04-30T06:42:48.710ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
33992025-04-30T06:42:48.712ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
34002025-04-30T06:42:48.731ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
34012025-04-30T06:42:48.741ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
34022025-04-30T06:42:48.742ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
34032025-04-30T06:42:48.742ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {} }
34042025-04-30T06:42:48.742ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
34052025-04-30T06:42:48.756ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
34062025-04-30T06:42:48.756ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {} }
34072025-04-30T06:42:48.764ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
34082025-04-30T06:42:48.841ZDEBGtest_snapshot_basic: attempt to fetch latest inventory collection result = Ok("found")
34092025-04-30T06:42:48.895ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
34102025-04-30T06:42:48.895ZTRACtest_snapshot_basic (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
34112025-04-30T06:42:48.895ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
34122025-04-30T06:42:48.895ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {} }
34132025-04-30T06:42:48.895ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
34142025-04-30T06:42:48.896ZWARNtest_snapshot_basic (datastore): Non-retryable transaction failure err = DatabaseError(UniqueViolation, "duplicate key value violates unique constraint \\"inv_caboose_pkey\\"") transaction = inventory_insert_collection
34152025-04-30T06:42:48.896ZWARNtest_snapshot_basic (BackgroundTasks): inventory collection failed background_task = inventory_collection error = failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint "inv_caboose_pkey"
34162025-04-30T06:42:48.896ZDEBGtest_snapshot_basic (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 804.790382ms iteration = 4 status = {"error":"failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint \\"inv_caboose_pkey\\""}
34172025-04-30T06:42:48.896ZDEBGtest_snapshot_basic (BackgroundTasks): activating background_task = inventory_collection iteration = 5 reason = Signaled
34182025-04-30T06:42:48.896ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
34192025-04-30T06:42:48.896ZTRACtest_snapshot_basic (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
34202025-04-30T06:42:48.896ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
34212025-04-30T06:42:48.896ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
34222025-04-30T06:42:48.897ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
34232025-04-30T06:42:48.899ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
34242025-04-30T06:42:48.913ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
34252025-04-30T06:42:48.913ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
34262025-04-30T06:42:48.913ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {} }
34272025-04-30T06:42:48.913ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
34282025-04-30T06:42:48.914ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
34292025-04-30T06:42:48.917ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
34302025-04-30T06:42:48.918ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
34312025-04-30T06:42:48.918ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
34322025-04-30T06:42:48.921ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
34332025-04-30T06:42:48.924ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
34342025-04-30T06:42:48.924ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {} }
34352025-04-30T06:42:48.924ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
34362025-04-30T06:42:48.925ZDEBGtest_snapshot_basic (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = [(d1ad44e5-18dd-4193-a70e-bf9042f0e0e1, 0)]
34372025-04-30T06:42:48.925ZTRACtest_snapshot_basic (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
34382025-04-30T06:42:48.927ZTRACtest_snapshot_basic (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: 2d 3a 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 -:..........._mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:62661 req_id = 09f7a84e-fab9-4730-b50d-d44cc28a06e8
34392025-04-30T06:42:48.927ZDEBGtest_snapshot_basic (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 11578,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:62661 req_id = 09f7a84e-fab9-4730-b50d-d44cc28a06e8
34402025-04-30T06:42:48.927ZDEBGtest_snapshot_basic (store): query key key = _mgs._tcp
34412025-04-30T06:42:48.927ZDEBGtest_snapshot_basic (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
34422025-04-30T06:42:48.927ZDEBGtest_snapshot_basic (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:62661 query = LowerQuery { name: LowerName(Name("_mgs._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 41502, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = 09f7a84e-fab9-4730-b50d-d44cc28a06e8
34432025-04-30T06:42:48.932ZDEBGtest_snapshot_basic (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 41502, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 4233, tv_nsec: 726923092 } })
34442025-04-30T06:42:48.932ZTRACtest_snapshot_basic (DnsResolver): lookup_service_targets: looking up SRV target
34452025-04-30T06:42:48.933ZTRACtest_snapshot_basic (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: fd 6f 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e .o..........-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:49727 req_id = 002d0714-8bf5-48dc-9826-64330c73d250
34462025-04-30T06:42:48.933ZDEBGtest_snapshot_basic (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 64879,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:49727 req_id = 002d0714-8bf5-48dc-9826-64330c73d250
34472025-04-30T06:42:48.933ZDEBGtest_snapshot_basic (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
34482025-04-30T06:42:48.933ZDEBGtest_snapshot_basic (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:49727 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = 002d0714-8bf5-48dc-9826-64330c73d250
34492025-04-30T06:42:48.985ZTRACtest_snapshot_basic (DnsResolver): lookup_all_socket_v6 srv dns_name = _clickhouse-admin-keeper._tcp.control-plane.oxide.internal
34502025-04-30T06:42:48.986ZTRACtest_snapshot_basic (dns): buffer buffer = Length: 87 (0x57) bytes\n0000: 81 9e 01 00 00 01 00 00 00 00 00 01 18 5f 63 6c ............._cl\n0010: 69 63 6b 68 6f 75 73 65 2d 61 64 6d 69 6e 2d 6b ickhouse-admin-k\n0020: 65 65 70 65 72 04 5f 74 63 70 0d 63 6f 6e 74 72 eeper._tcp.contr\n0030: 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 69 ol-plane.oxide.i\n0040: 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 04 nternal..!....).\n0050: d0 00 00 00 00 00 00 ....... kind = dns peer_addr = [::1]:52133 req_id = 1ebeeff9-1ac8-4d81-b74f-f664e9bed42d
34512025-04-30T06:42:48.986ZDEBGtest_snapshot_basic (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 33182,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 24,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 97,\n 100,\n 109,\n 105,\n 110,\n 45,\n 107,\n 101,\n 101,\n 112,\n 101,\n 114,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:52133 req_id = 1ebeeff9-1ac8-4d81-b74f-f664e9bed42d
34522025-04-30T06:42:48.986ZDEBGtest_snapshot_basic (store): query key key = _clickhouse-admin-keeper._tcp
34532025-04-30T06:42:48.987ZERROtest_snapshot_basic (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 33182, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } NXDOMAIN: no records found for name: "_clickhouse-admin-keeper._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:52133 req_id = 1ebeeff9-1ac8-4d81-b74f-f664e9bed42d
34542025-04-30T06:42:48.987ZDEBGtest_snapshot_basic (BackgroundTasks): begin collection background_task = inventory_collection
34552025-04-30T06:42:48.987ZDEBGtest_snapshot_basic (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:41502
34562025-04-30T06:42:48.987ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/ignition
34572025-04-30T06:42:48.994ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "21d06264-c16c-434c-9c21-d20a8e6f20e0", "content-length": "835", "date": "Wed, 30 Apr 2025 06:42:48 GMT"} })
34582025-04-30T06:42:48.994ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/0
34592025-04-30T06:42:48.996ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "eb4f24ee-8513-4a95-bb74-d07d0c9b8e1b", "content-length": "734", "date": "Wed, 30 Apr 2025 06:42:48 GMT"} })
34602025-04-30T06:42:48.997ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/0/component/sp/caboose?firmware_slot=0
34612025-04-30T06:42:49.006ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "d8e94540-beb1-40f7-9b14-4b77dcaa7943", "content-length": "111", "date": "Wed, 30 Apr 2025 06:42:48 GMT"} })
34622025-04-30T06:42:49.006ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/0/component/sp/caboose?firmware_slot=1
34632025-04-30T06:42:49.008ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "6d34e025-914c-4465-91f8-5b1860564592", "content-length": "111", "date": "Wed, 30 Apr 2025 06:42:48 GMT"} })
34642025-04-30T06:42:49.008ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/0/component/rot/caboose?firmware_slot=0
34652025-04-30T06:42:49.014ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "4dd9238e-ec22-461d-96bf-96de3419add7", "content-length": "170", "date": "Wed, 30 Apr 2025 06:42:48 GMT"} })
34662025-04-30T06:42:49.015ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/0/component/rot/caboose?firmware_slot=1
34672025-04-30T06:42:49.027ZDEBGtest_snapshot_basic: attempt to fetch latest inventory collection result = Ok("found")
34682025-04-30T06:42:49.030ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "261fdaad-6991-4337-98bf-40e8add08ccc", "content-length": "170", "date": "Wed, 30 Apr 2025 06:42:48 GMT"} })
34692025-04-30T06:42:49.030ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/0/component/stage0/caboose?firmware_slot=0
34702025-04-30T06:42:49.033ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "e00e47b0-cd8d-4ee6-bc3d-cc96bdf503fc", "content-length": "179", "date": "Wed, 30 Apr 2025 06:42:48 GMT"} })
34712025-04-30T06:42:49.033ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/0/component/stage0/caboose?firmware_slot=1
34722025-04-30T06:42:49.034ZTRACtest_snapshot_basic (results-sink): batch interval expired, but no samples to insert collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
34732025-04-30T06:42:49.034ZTRACtest_snapshot_basic (results-sink-cluster): batch interval expired, but no samples to insert collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
34742025-04-30T06:42:49.036ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "06e8ae41-438b-4569-8923-1371c0158e94", "content-length": "179", "date": "Wed, 30 Apr 2025 06:42:48 GMT"} })
34752025-04-30T06:42:49.036ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/0/component/rot/cmpa
34762025-04-30T06:42:49.037ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "243b7f60-1ec9-4620-889f-00bc47cac242", "content-length": "702", "date": "Wed, 30 Apr 2025 06:42:48 GMT"} })
34772025-04-30T06:42:49.037ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/0/component/rot/cfpa
34782025-04-30T06:42:49.040ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "4636a91c-7ea0-47e2-bd5e-c77260baa800", "content-length": "727", "date": "Wed, 30 Apr 2025 06:42:48 GMT"} })
34792025-04-30T06:42:49.040ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/0/component/rot/cfpa
34802025-04-30T06:42:49.042ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "b20d900a-549d-4c9f-91c9-e3680ebfa43c", "content-length": "729", "date": "Wed, 30 Apr 2025 06:42:48 GMT"} })
34812025-04-30T06:42:49.043ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/0/component/rot/cfpa
34822025-04-30T06:42:49.045ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "a1797723-9e9e-40a3-9720-449e25393029", "content-length": "728", "date": "Wed, 30 Apr 2025 06:42:48 GMT"} })
34832025-04-30T06:42:49.045ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/1
34842025-04-30T06:42:49.048ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "23168091-b040-4dae-8379-b08859014431", "content-length": "734", "date": "Wed, 30 Apr 2025 06:42:48 GMT"} })
34852025-04-30T06:42:49.049ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/1/component/sp/caboose?firmware_slot=0
34862025-04-30T06:42:49.051ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "b7435990-ef35-4a18-be28-993679d74b2d", "content-length": "111", "date": "Wed, 30 Apr 2025 06:42:48 GMT"} })
34872025-04-30T06:42:49.051ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/1/component/sp/caboose?firmware_slot=1
34882025-04-30T06:42:49.053ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "88166c47-f759-4904-9303-ea4a5bc8c50d", "content-length": "111", "date": "Wed, 30 Apr 2025 06:42:48 GMT"} })
34892025-04-30T06:42:49.053ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/1/component/rot/caboose?firmware_slot=0
34902025-04-30T06:42:49.056ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "6d5159d3-70bf-4f9a-96b5-3de40ab289c3", "content-length": "170", "date": "Wed, 30 Apr 2025 06:42:48 GMT"} })
34912025-04-30T06:42:49.056ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/1/component/rot/caboose?firmware_slot=1
34922025-04-30T06:42:49.060ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "aedcd330-871e-4fd0-9624-df8d068a1b67", "content-length": "170", "date": "Wed, 30 Apr 2025 06:42:48 GMT"} })
34932025-04-30T06:42:49.060ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/1/component/stage0/caboose?firmware_slot=0
34942025-04-30T06:42:49.065ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "bc3b287c-fb81-467d-ba21-f924986e8abf", "content-length": "179", "date": "Wed, 30 Apr 2025 06:42:48 GMT"} })
34952025-04-30T06:42:49.065ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/1/component/stage0/caboose?firmware_slot=1
34962025-04-30T06:42:49.066ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "e5f9257c-d545-49c1-86c0-cb21a6b021e8", "content-length": "179", "date": "Wed, 30 Apr 2025 06:42:48 GMT"} })
34972025-04-30T06:42:49.067ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/1/component/rot/cmpa
34982025-04-30T06:42:49.067ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "95add35a-3e34-427e-8c8a-96108c6479fc", "content-length": "702", "date": "Wed, 30 Apr 2025 06:42:48 GMT"} })
34992025-04-30T06:42:49.068ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/1/component/rot/cfpa
35002025-04-30T06:42:49.069ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "ba45861d-e867-4604-a345-67b0d7a282fb", "content-length": "727", "date": "Wed, 30 Apr 2025 06:42:48 GMT"} })
35012025-04-30T06:42:49.069ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/1/component/rot/cfpa
35022025-04-30T06:42:49.070ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "e13e84ad-8724-426a-bab6-f30bec782f74", "content-length": "729", "date": "Wed, 30 Apr 2025 06:42:48 GMT"} })
35032025-04-30T06:42:49.070ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/1/component/rot/cfpa
35042025-04-30T06:42:49.071ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "3023bb78-21ca-4073-bf53-a84b97744489", "content-length": "728", "date": "Wed, 30 Apr 2025 06:42:48 GMT"} })
35052025-04-30T06:42:49.071ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/0
35062025-04-30T06:42:49.072ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "0f3fed7d-b361-4ac8-a5a6-5fb55c0af7f6", "content-length": "723", "date": "Wed, 30 Apr 2025 06:42:48 GMT"} })
35072025-04-30T06:42:49.072ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/0/component/sp/caboose?firmware_slot=0
35082025-04-30T06:42:49.075ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "c90b8bc1-e3a1-4515-83eb-fdc7ee0099f6", "content-length": "109", "date": "Wed, 30 Apr 2025 06:42:48 GMT"} })
35092025-04-30T06:42:49.075ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/0/component/sp/caboose?firmware_slot=1
35102025-04-30T06:42:49.077ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "ea8e3525-b79f-46b6-9266-6ef031176e20", "content-length": "109", "date": "Wed, 30 Apr 2025 06:42:48 GMT"} })
35112025-04-30T06:42:49.077ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/0/component/rot/caboose?firmware_slot=0
35122025-04-30T06:42:49.078ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
35132025-04-30T06:42:49.078ZTRACtest_snapshot_basic (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
35142025-04-30T06:42:49.078ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
35152025-04-30T06:42:49.078ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {} }
35162025-04-30T06:42:49.079ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
35172025-04-30T06:42:49.081ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
35182025-04-30T06:42:49.086ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
35192025-04-30T06:42:49.087ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
35202025-04-30T06:42:49.087ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {} }
35212025-04-30T06:42:49.087ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
35222025-04-30T06:42:49.088ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "9699dd8a-e95c-4a5e-899f-4df781c87dd9", "content-length": "169", "date": "Wed, 30 Apr 2025 06:42:48 GMT"} })
35232025-04-30T06:42:49.088ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/0/component/rot/caboose?firmware_slot=1
35242025-04-30T06:42:49.090ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
35252025-04-30T06:42:49.090ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {} }
35262025-04-30T06:42:49.091ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
35272025-04-30T06:42:49.099ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "ae2a6c56-39fd-4f1a-9446-bada7deb253c", "content-length": "169", "date": "Wed, 30 Apr 2025 06:42:48 GMT"} })
35282025-04-30T06:42:49.100ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/0/component/stage0/caboose?firmware_slot=0
35292025-04-30T06:42:49.102ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "c1677de3-6c71-4b6b-ad1d-03b9d47d7c47", "content-length": "178", "date": "Wed, 30 Apr 2025 06:42:48 GMT"} })
35302025-04-30T06:42:49.103ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/0/component/stage0/caboose?firmware_slot=1
35312025-04-30T06:42:49.115ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "253d0778-6c79-401e-ae0b-335438bac69c", "content-length": "178", "date": "Wed, 30 Apr 2025 06:42:48 GMT"} })
35322025-04-30T06:42:49.116ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/0/component/rot/cmpa
35332025-04-30T06:42:49.122ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "3ec8d569-1256-4a06-894a-ee7e2cdef4b1", "content-length": "702", "date": "Wed, 30 Apr 2025 06:42:48 GMT"} })
35342025-04-30T06:42:49.122ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/0/component/rot/cfpa
35352025-04-30T06:42:49.124ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "59392be4-35d0-4462-8256-e2ba9f5adb10", "content-length": "727", "date": "Wed, 30 Apr 2025 06:42:48 GMT"} })
35362025-04-30T06:42:49.124ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/0/component/rot/cfpa
35372025-04-30T06:42:49.126ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "5aeaf56e-babe-432f-9105-e0f96b9bbd59", "content-length": "729", "date": "Wed, 30 Apr 2025 06:42:48 GMT"} })
35382025-04-30T06:42:49.126ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/0/component/rot/cfpa
35392025-04-30T06:42:49.127ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "297bb6d2-296f-453c-a74b-a83e604780fb", "content-length": "728", "date": "Wed, 30 Apr 2025 06:42:48 GMT"} })
35402025-04-30T06:42:49.127ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/1
35412025-04-30T06:42:49.135ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "dd4b887d-a4d1-43ef-9553-7d7ca3a13ac4", "content-length": "723", "date": "Wed, 30 Apr 2025 06:42:48 GMT"} })
35422025-04-30T06:42:49.135ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/1/component/sp/caboose?firmware_slot=0
35432025-04-30T06:42:49.142ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "4c7c8813-73a4-4094-b10b-947bf3da95fb", "content-length": "109", "date": "Wed, 30 Apr 2025 06:42:48 GMT"} })
35442025-04-30T06:42:49.143ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/1/component/sp/caboose?firmware_slot=1
35452025-04-30T06:42:49.144ZDEBGtest_snapshot_basic: attempt to fetch latest inventory collection result = Ok("found")
35462025-04-30T06:42:49.149ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "7dae82db-74cc-4217-b2dc-24df2cbfea6c", "content-length": "109", "date": "Wed, 30 Apr 2025 06:42:48 GMT"} })
35472025-04-30T06:42:49.149ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/1/component/rot/caboose?firmware_slot=0
35482025-04-30T06:42:49.152ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "da34d08f-251d-4d9d-9cc2-608d2a629591", "content-length": "169", "date": "Wed, 30 Apr 2025 06:42:48 GMT"} })
35492025-04-30T06:42:49.152ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/1/component/rot/caboose?firmware_slot=1
35502025-04-30T06:42:49.154ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "6d0d40df-b504-4323-818d-ec0a955e5898", "content-length": "169", "date": "Wed, 30 Apr 2025 06:42:48 GMT"} })
35512025-04-30T06:42:49.154ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/1/component/stage0/caboose?firmware_slot=0
35522025-04-30T06:42:49.156ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "f30e4a1f-cfc8-4202-9474-f6bbbcd5c77e", "content-length": "178", "date": "Wed, 30 Apr 2025 06:42:48 GMT"} })
35532025-04-30T06:42:49.156ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/1/component/stage0/caboose?firmware_slot=1
35542025-04-30T06:42:49.159ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "d608e624-cc92-4abc-9415-9c4a03c1cfb5", "content-length": "178", "date": "Wed, 30 Apr 2025 06:42:48 GMT"} })
35552025-04-30T06:42:49.159ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/1/component/rot/cmpa
35562025-04-30T06:42:49.160ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "ad461af8-6a39-4dac-83ec-d0ad0bfd1552", "content-length": "702", "date": "Wed, 30 Apr 2025 06:42:48 GMT"} })
35572025-04-30T06:42:49.160ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/1/component/rot/cfpa
35582025-04-30T06:42:49.161ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "b5a5ff84-bed3-47ed-9dd2-e9c3f3c3b670", "content-length": "727", "date": "Wed, 30 Apr 2025 06:42:48 GMT"} })
35592025-04-30T06:42:49.161ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/1/component/rot/cfpa
35602025-04-30T06:42:49.162ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "555fd732-f1b5-43f1-8226-03e2a1a0f1ba", "content-length": "729", "date": "Wed, 30 Apr 2025 06:42:48 GMT"} })
35612025-04-30T06:42:49.162ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/1/component/rot/cfpa
35622025-04-30T06:42:49.163ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "6a6318fa-9d79-4964-a4a1-0a332e848005", "content-length": "728", "date": "Wed, 30 Apr 2025 06:42:48 GMT"} })
35632025-04-30T06:42:49.163ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
35642025-04-30T06:42:49.164ZTRACtest_snapshot_basic (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
35652025-04-30T06:42:49.166ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
35662025-04-30T06:42:49.166ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
35672025-04-30T06:42:49.167ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
35682025-04-30T06:42:49.175ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
35692025-04-30T06:42:49.181ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
35702025-04-30T06:42:49.181ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
35712025-04-30T06:42:49.181ZTRACtest_snapshot_basic (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
35722025-04-30T06:42:49.181ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
35732025-04-30T06:42:49.181ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
35742025-04-30T06:42:49.182ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
35752025-04-30T06:42:49.185ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
35762025-04-30T06:42:49.190ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
35772025-04-30T06:42:49.190ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
35782025-04-30T06:42:49.190ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
35792025-04-30T06:42:49.190ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
35802025-04-30T06:42:49.196ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
35812025-04-30T06:42:49.196ZTRACtest_snapshot_basic (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
35822025-04-30T06:42:49.196ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
35832025-04-30T06:42:49.196ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {} }
35842025-04-30T06:42:49.197ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
35852025-04-30T06:42:49.266ZDEBGtest_snapshot_basic (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:41923
35862025-04-30T06:42:49.266ZDEBGtest_snapshot_basic (BackgroundTasks): client request SledAgent = http://[::1]:41923 background_task = inventory_collection body = None method = GET uri = http://[::1]:41923/inventory
35872025-04-30T06:42:49.266ZINFOtest_snapshot_basic (dropshot): accepted connection local_addr = [::1]:41923 remote_addr = [::1]:38141 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
35882025-04-30T06:42:49.271ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
35892025-04-30T06:42:49.292ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
35902025-04-30T06:42:49.293ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
35912025-04-30T06:42:49.293ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {} }
35922025-04-30T06:42:49.294ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
35932025-04-30T06:42:49.297ZTRACtest_snapshot_basic (dropshot): incoming request local_addr = [::1]:41923 method = GET remote_addr = [::1]:38141 req_id = 2d9a3c64-0149-4d8a-a037-3f9c0fbcf07c sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
35942025-04-30T06:42:49.301ZINFOtest_snapshot_basic (dropshot): request completed latency_us = 6319 local_addr = [::1]:41923 method = GET remote_addr = [::1]:38141 req_id = 2d9a3c64-0149-4d8a-a037-3f9c0fbcf07c response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
35952025-04-30T06:42:49.308ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
35962025-04-30T06:42:49.308ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {} }
35972025-04-30T06:42:49.309ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
35982025-04-30T06:42:49.309ZDEBGtest_snapshot_basic (BackgroundTasks): client response SledAgent = http://[::1]:41923 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:41923/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "2d9a3c64-0149-4d8a-a037-3f9c0fbcf07c", "content-length": "3141", "date": "Wed, 30 Apr 2025 06:42:48 GMT"} })
35992025-04-30T06:42:49.311ZDEBGtest_snapshot_basic (BackgroundTasks): begin collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
36002025-04-30T06:42:49.311ZDEBGtest_snapshot_basic (BackgroundTasks): end collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
36012025-04-30T06:42:49.311ZDEBGtest_snapshot_basic (BackgroundTasks): finished collection background_task = inventory_collection
36022025-04-30T06:42:49.313ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
36032025-04-30T06:42:49.314ZTRACtest_snapshot_basic (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
36042025-04-30T06:42:49.314ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
36052025-04-30T06:42:49.314ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
36062025-04-30T06:42:49.314ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
36072025-04-30T06:42:49.320ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
36082025-04-30T06:42:49.327ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
36092025-04-30T06:42:49.328ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
36102025-04-30T06:42:49.329ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
36112025-04-30T06:42:49.330ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
36122025-04-30T06:42:49.390ZDEBGtest_snapshot_basic: attempt to fetch latest inventory collection result = Ok("found")
36132025-04-30T06:42:49.447ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
36142025-04-30T06:42:49.448ZTRACtest_snapshot_basic (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
36152025-04-30T06:42:49.448ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
36162025-04-30T06:42:49.448ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {} }
36172025-04-30T06:42:49.451ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
36182025-04-30T06:42:49.458ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
36192025-04-30T06:42:49.473ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
36202025-04-30T06:42:49.473ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
36212025-04-30T06:42:49.473ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {} }
36222025-04-30T06:42:49.473ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
36232025-04-30T06:42:49.492ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
36242025-04-30T06:42:49.492ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {} }
36252025-04-30T06:42:49.493ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
36262025-04-30T06:42:49.493ZWARNtest_snapshot_basic (datastore): Non-retryable transaction failure err = DatabaseError(UniqueViolation, "duplicate key value violates unique constraint \\"inv_caboose_pkey\\"") transaction = inventory_insert_collection
36272025-04-30T06:42:49.494ZWARNtest_snapshot_basic (BackgroundTasks): inventory collection failed background_task = inventory_collection error = failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint "inv_caboose_pkey"
36282025-04-30T06:42:49.494ZDEBGtest_snapshot_basic (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 597.458728ms iteration = 5 status = {"error":"failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint \\"inv_caboose_pkey\\""}
36292025-04-30T06:42:49.494ZDEBGtest_snapshot_basic (BackgroundTasks): activating background_task = inventory_collection iteration = 6 reason = Signaled
36302025-04-30T06:42:49.494ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
36312025-04-30T06:42:49.494ZTRACtest_snapshot_basic (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
36322025-04-30T06:42:49.494ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
36332025-04-30T06:42:49.494ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
36342025-04-30T06:42:49.494ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
36352025-04-30T06:42:49.497ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
36362025-04-30T06:42:49.506ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
36372025-04-30T06:42:49.506ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
36382025-04-30T06:42:49.507ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
36392025-04-30T06:42:49.508ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
36402025-04-30T06:42:49.517ZDEBGtest_snapshot_basic (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = [(d1ad44e5-18dd-4193-a70e-bf9042f0e0e1, 0)]
36412025-04-30T06:42:49.517ZTRACtest_snapshot_basic (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
36422025-04-30T06:42:49.524ZTRACtest_snapshot_basic (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: 9d 92 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 ............._mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:64290 req_id = 9d5c3001-a425-451f-b241-343fb9f85cf8
36432025-04-30T06:42:49.524ZDEBGtest_snapshot_basic (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 40338,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:64290 req_id = 9d5c3001-a425-451f-b241-343fb9f85cf8
36442025-04-30T06:42:49.524ZDEBGtest_snapshot_basic (store): query key key = _mgs._tcp
36452025-04-30T06:42:49.524ZDEBGtest_snapshot_basic (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
36462025-04-30T06:42:49.524ZDEBGtest_snapshot_basic (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:64290 query = LowerQuery { name: LowerName(Name("_mgs._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 41502, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = 9d5c3001-a425-451f-b241-343fb9f85cf8
36472025-04-30T06:42:49.525ZDEBGtest_snapshot_basic (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 41502, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 4234, tv_nsec: 324200017 } })
36482025-04-30T06:42:49.525ZTRACtest_snapshot_basic (DnsResolver): lookup_service_targets: looking up SRV target
36492025-04-30T06:42:49.527ZTRACtest_snapshot_basic (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: bc bb 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e ............-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:49577 req_id = 52814aa4-6b3a-4887-971b-c0a407735d76
36502025-04-30T06:42:49.527ZDEBGtest_snapshot_basic (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 48315,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:49577 req_id = 52814aa4-6b3a-4887-971b-c0a407735d76
36512025-04-30T06:42:49.527ZDEBGtest_snapshot_basic (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
36522025-04-30T06:42:49.527ZDEBGtest_snapshot_basic (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:49577 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = 52814aa4-6b3a-4887-971b-c0a407735d76
36532025-04-30T06:42:49.594ZTRACtest_snapshot_basic (DnsResolver): lookup_all_socket_v6 srv dns_name = _clickhouse-admin-keeper._tcp.control-plane.oxide.internal
36542025-04-30T06:42:49.605ZTRACtest_snapshot_basic (dns): buffer buffer = Length: 87 (0x57) bytes\n0000: 1f 29 01 00 00 01 00 00 00 00 00 01 18 5f 63 6c .)..........._cl\n0010: 69 63 6b 68 6f 75 73 65 2d 61 64 6d 69 6e 2d 6b ickhouse-admin-k\n0020: 65 65 70 65 72 04 5f 74 63 70 0d 63 6f 6e 74 72 eeper._tcp.contr\n0030: 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 69 ol-plane.oxide.i\n0040: 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 04 nternal..!....).\n0050: d0 00 00 00 00 00 00 ....... kind = dns peer_addr = [::1]:51500 req_id = ce91423c-80ec-4e9b-935e-655df74bf29a
36552025-04-30T06:42:49.605ZDEBGtest_snapshot_basic (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 7977,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 24,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 97,\n 100,\n 109,\n 105,\n 110,\n 45,\n 107,\n 101,\n 101,\n 112,\n 101,\n 114,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:51500 req_id = ce91423c-80ec-4e9b-935e-655df74bf29a
36562025-04-30T06:42:49.605ZDEBGtest_snapshot_basic (store): query key key = _clickhouse-admin-keeper._tcp
36572025-04-30T06:42:49.606ZERROtest_snapshot_basic (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 7977, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } NXDOMAIN: no records found for name: "_clickhouse-admin-keeper._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:51500 req_id = ce91423c-80ec-4e9b-935e-655df74bf29a
36582025-04-30T06:42:49.607ZDEBGtest_snapshot_basic (BackgroundTasks): begin collection background_task = inventory_collection
36592025-04-30T06:42:49.607ZDEBGtest_snapshot_basic (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:41502
36602025-04-30T06:42:49.607ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/ignition
36612025-04-30T06:42:49.627ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "6281d8c3-c50f-448f-8b78-790b0d8fecec", "content-length": "835", "date": "Wed, 30 Apr 2025 06:42:48 GMT"} })
36622025-04-30T06:42:49.627ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/0
36632025-04-30T06:42:49.629ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "61f5bb53-ea56-421f-af78-a78c88bd5c95", "content-length": "734", "date": "Wed, 30 Apr 2025 06:42:48 GMT"} })
36642025-04-30T06:42:49.630ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/0/component/sp/caboose?firmware_slot=0
36652025-04-30T06:42:49.632ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "1851ea48-abee-498b-a3b2-66d8b2635444", "content-length": "111", "date": "Wed, 30 Apr 2025 06:42:48 GMT"} })
36662025-04-30T06:42:49.632ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/0/component/sp/caboose?firmware_slot=1
36672025-04-30T06:42:49.635ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "36943838-9d39-4b99-a7bc-d4e831f9fd1c", "content-length": "111", "date": "Wed, 30 Apr 2025 06:42:48 GMT"} })
36682025-04-30T06:42:49.635ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/0/component/rot/caboose?firmware_slot=0
36692025-04-30T06:42:49.638ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "f78c0fa9-166e-4b90-ab71-45973d7f6db9", "content-length": "170", "date": "Wed, 30 Apr 2025 06:42:48 GMT"} })
36702025-04-30T06:42:49.638ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/0/component/rot/caboose?firmware_slot=1
36712025-04-30T06:42:49.647ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "bc36a409-90ee-4914-b4a7-e347d8d477ae", "content-length": "170", "date": "Wed, 30 Apr 2025 06:42:48 GMT"} })
36722025-04-30T06:42:49.647ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/0/component/stage0/caboose?firmware_slot=0
36732025-04-30T06:42:49.648ZDEBGtest_snapshot_basic: attempt to fetch latest inventory collection result = Ok("found")
36742025-04-30T06:42:49.651ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "7f40d5eb-075f-4f8e-a8db-6385bfecba83", "content-length": "179", "date": "Wed, 30 Apr 2025 06:42:48 GMT"} })
36752025-04-30T06:42:49.651ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/0/component/stage0/caboose?firmware_slot=1
36762025-04-30T06:42:49.653ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "e25c3328-93fe-4d6b-b059-d921971f4a20", "content-length": "179", "date": "Wed, 30 Apr 2025 06:42:48 GMT"} })
36772025-04-30T06:42:49.653ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/0/component/rot/cmpa
36782025-04-30T06:42:49.654ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "35fecef1-e249-4cc4-8e68-ce4478caf1e9", "content-length": "702", "date": "Wed, 30 Apr 2025 06:42:48 GMT"} })
36792025-04-30T06:42:49.655ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/0/component/rot/cfpa
36802025-04-30T06:42:49.657ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "0f2c3e48-d6ab-4308-a950-5c813a449e5f", "content-length": "727", "date": "Wed, 30 Apr 2025 06:42:48 GMT"} })
36812025-04-30T06:42:49.657ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/0/component/rot/cfpa
36822025-04-30T06:42:49.658ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "8aff99e4-dd02-4cec-8265-b14180ded0c7", "content-length": "729", "date": "Wed, 30 Apr 2025 06:42:48 GMT"} })
36832025-04-30T06:42:49.659ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/0/component/rot/cfpa
36842025-04-30T06:42:49.660ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "fc85a941-989b-44ab-8759-d818cc90924b", "content-length": "728", "date": "Wed, 30 Apr 2025 06:42:48 GMT"} })
36852025-04-30T06:42:49.660ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/1
36862025-04-30T06:42:49.664ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "ce7e8d03-4da9-4ce3-bf92-d3c0322bb93b", "content-length": "734", "date": "Wed, 30 Apr 2025 06:42:48 GMT"} })
36872025-04-30T06:42:49.665ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/1/component/sp/caboose?firmware_slot=0
36882025-04-30T06:42:49.672ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "492a3af5-4af8-4f0e-909e-bee45647618f", "content-length": "111", "date": "Wed, 30 Apr 2025 06:42:48 GMT"} })
36892025-04-30T06:42:49.673ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/1/component/sp/caboose?firmware_slot=1
36902025-04-30T06:42:49.674ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "9a8e0291-ee30-45b9-aa73-1a1cfd51de67", "content-length": "111", "date": "Wed, 30 Apr 2025 06:42:48 GMT"} })
36912025-04-30T06:42:49.674ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/1/component/rot/caboose?firmware_slot=0
36922025-04-30T06:42:49.677ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "a224455d-89bc-43af-88f5-600f95ea7279", "content-length": "170", "date": "Wed, 30 Apr 2025 06:42:48 GMT"} })
36932025-04-30T06:42:49.677ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/1/component/rot/caboose?firmware_slot=1
36942025-04-30T06:42:49.679ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "2c1a518e-9d68-400a-a42b-c8703775cb70", "content-length": "170", "date": "Wed, 30 Apr 2025 06:42:48 GMT"} })
36952025-04-30T06:42:49.679ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/1/component/stage0/caboose?firmware_slot=0
36962025-04-30T06:42:49.681ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "478babd9-6b7e-4607-9bf1-961a52e277d5", "content-length": "179", "date": "Wed, 30 Apr 2025 06:42:48 GMT"} })
36972025-04-30T06:42:49.681ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/1/component/stage0/caboose?firmware_slot=1
36982025-04-30T06:42:49.683ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "7a24239e-f68c-4a11-810e-ff01e3c7e090", "content-length": "179", "date": "Wed, 30 Apr 2025 06:42:48 GMT"} })
36992025-04-30T06:42:49.683ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/1/component/rot/cmpa
37002025-04-30T06:42:49.684ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "162707c3-d4bb-44fe-ad7f-cdb456570905", "content-length": "702", "date": "Wed, 30 Apr 2025 06:42:48 GMT"} })
37012025-04-30T06:42:49.684ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/1/component/rot/cfpa
37022025-04-30T06:42:49.687ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "8b7f018d-8a46-4b6f-99a8-8b5a2fb48d9a", "content-length": "727", "date": "Wed, 30 Apr 2025 06:42:48 GMT"} })
37032025-04-30T06:42:49.687ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/1/component/rot/cfpa
37042025-04-30T06:42:49.688ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "961b6a21-1195-479e-9f02-682085d9ab93", "content-length": "729", "date": "Wed, 30 Apr 2025 06:42:48 GMT"} })
37052025-04-30T06:42:49.688ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/1/component/rot/cfpa
37062025-04-30T06:42:49.689ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "9903bb5d-81ed-4056-9608-24c655e46f73", "content-length": "728", "date": "Wed, 30 Apr 2025 06:42:48 GMT"} })
37072025-04-30T06:42:49.689ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/0
37082025-04-30T06:42:49.691ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "332ce76c-3706-4cc7-8b64-478d7263396e", "content-length": "723", "date": "Wed, 30 Apr 2025 06:42:48 GMT"} })
37092025-04-30T06:42:49.691ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/0/component/sp/caboose?firmware_slot=0
37102025-04-30T06:42:49.693ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "05652ad8-dae1-4ec5-9376-cfca1a92d01a", "content-length": "109", "date": "Wed, 30 Apr 2025 06:42:48 GMT"} })
37112025-04-30T06:42:49.693ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/0/component/sp/caboose?firmware_slot=1
37122025-04-30T06:42:49.695ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "2fab97e6-97de-41bd-9060-4255b933ab4d", "content-length": "109", "date": "Wed, 30 Apr 2025 06:42:48 GMT"} })
37132025-04-30T06:42:49.695ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/0/component/rot/caboose?firmware_slot=0
37142025-04-30T06:42:49.698ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "0623f4f2-5b60-4acf-a9f1-cea1c0ff9c44", "content-length": "169", "date": "Wed, 30 Apr 2025 06:42:48 GMT"} })
37152025-04-30T06:42:49.698ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/0/component/rot/caboose?firmware_slot=1
37162025-04-30T06:42:49.698ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
37172025-04-30T06:42:49.698ZTRACtest_snapshot_basic (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
37182025-04-30T06:42:49.698ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
37192025-04-30T06:42:49.698ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {} }
37202025-04-30T06:42:49.700ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
37212025-04-30T06:42:49.704ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "90c3f9ea-91a9-46d3-8d0b-8044067a1310", "content-length": "169", "date": "Wed, 30 Apr 2025 06:42:48 GMT"} })
37222025-04-30T06:42:49.705ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/0/component/stage0/caboose?firmware_slot=0
37232025-04-30T06:42:49.706ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
37242025-04-30T06:42:49.714ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
37252025-04-30T06:42:49.715ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
37262025-04-30T06:42:49.715ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {} }
37272025-04-30T06:42:49.715ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
37282025-04-30T06:42:49.719ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
37292025-04-30T06:42:49.719ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {} }
37302025-04-30T06:42:49.720ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
37312025-04-30T06:42:49.721ZINFOtest_snapshot_basic (oximeter-agent): refreshing list of producers from Nexus collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 file = oximeter/collector/src/agent.rs:562
37322025-04-30T06:42:49.722ZDEBGtest_snapshot_basic (oximeter): client request body = None method = GET uri = http://127.0.0.1:61468/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
37332025-04-30T06:42:49.722ZTRACtest_snapshot_basic (dropshot_internal): incoming request local_addr = 127.0.0.1:61468 method = GET remote_addr = 127.0.0.1:64332 req_id = a3e80ee8-ab65-4617-888e-c70b75cec111 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
37342025-04-30T06:42:49.723ZTRACtest_snapshot_basic (dropshot_internal): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:61468 method = GET remote_addr = 127.0.0.1:64332 req_id = a3e80ee8-ab65-4617-888e-c70b75cec111 resource = Database uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
37352025-04-30T06:42:49.723ZDEBGtest_snapshot_basic (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:61468 method = GET remote_addr = 127.0.0.1:64332 req_id = a3e80ee8-ab65-4617-888e-c70b75cec111 roles = RoleSet { roles: {} } uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
37362025-04-30T06:42:49.724ZDEBGtest_snapshot_basic (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:61468 method = GET remote_addr = 127.0.0.1:64332 req_id = a3e80ee8-ab65-4617-888e-c70b75cec111 resource = Database result = Ok(()) uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
37372025-04-30T06:42:49.729ZINFOtest_snapshot_basic (dropshot_internal): request completed latency_us = 6626 local_addr = 127.0.0.1:61468 method = GET remote_addr = 127.0.0.1:64332 req_id = a3e80ee8-ab65-4617-888e-c70b75cec111 response_code = 200 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
37382025-04-30T06:42:49.731ZDEBGtest_snapshot_basic (oximeter): client response result = Ok(Response { url: "http://127.0.0.1:61468/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending", status: 200, headers: {"content-type": "application/json", "x-request-id": "a3e80ee8-ab65-4617-888e-c70b75cec111", "content-length": "405", "date": "Wed, 30 Apr 2025 06:42:48 GMT"} })
37392025-04-30T06:42:49.732ZDEBGtest_snapshot_basic (oximeter): client request body = None method = GET uri = http://127.0.0.1:61468/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
37402025-04-30T06:42:49.732ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "8ec881cf-9988-429d-940e-fb90d5900263", "content-length": "178", "date": "Wed, 30 Apr 2025 06:42:48 GMT"} })
37412025-04-30T06:42:49.732ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/0/component/stage0/caboose?firmware_slot=1
37422025-04-30T06:42:49.733ZTRACtest_snapshot_basic (dropshot_internal): incoming request local_addr = 127.0.0.1:61468 method = GET remote_addr = 127.0.0.1:64332 req_id = bd148792-4ae5-4d53-bda8-984478c7efba uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
37432025-04-30T06:42:49.736ZTRACtest_snapshot_basic (dropshot_internal): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:61468 method = GET remote_addr = 127.0.0.1:64332 req_id = bd148792-4ae5-4d53-bda8-984478c7efba resource = Database uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
37442025-04-30T06:42:49.736ZDEBGtest_snapshot_basic (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:61468 method = GET remote_addr = 127.0.0.1:64332 req_id = bd148792-4ae5-4d53-bda8-984478c7efba roles = RoleSet { roles: {} } uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
37452025-04-30T06:42:49.738ZDEBGtest_snapshot_basic (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:61468 method = GET remote_addr = 127.0.0.1:64332 req_id = bd148792-4ae5-4d53-bda8-984478c7efba resource = Database result = Ok(()) uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
37462025-04-30T06:42:49.741ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "c335088b-992b-4006-a97f-759a5f812056", "content-length": "178", "date": "Wed, 30 Apr 2025 06:42:48 GMT"} })
37472025-04-30T06:42:49.741ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/0/component/rot/cmpa
37482025-04-30T06:42:49.742ZINFOtest_snapshot_basic (dropshot_internal): request completed latency_us = 8951 local_addr = 127.0.0.1:61468 method = GET remote_addr = 127.0.0.1:64332 req_id = bd148792-4ae5-4d53-bda8-984478c7efba response_code = 200 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
37492025-04-30T06:42:49.742ZDEBGtest_snapshot_basic (oximeter): client response result = Ok(Response { url: "http://127.0.0.1:61468/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D", status: 200, headers: {"content-type": "application/json", "x-request-id": "bd148792-4ae5-4d53-bda8-984478c7efba", "content-length": "29", "date": "Wed, 30 Apr 2025 06:42:48 GMT"} })
37502025-04-30T06:42:49.742ZDEBGtest_snapshot_basic (oximeter-agent): registered new metric producer address = 127.0.0.1:47319 collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = 354a52d4-3078-4706-81ce-12c32f6bbeaa
37512025-04-30T06:42:49.792ZTRACtest_snapshot_basic (collection-task): top of inner collection loop, waiting for next request collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = 354a52d4-3078-4706-81ce-12c32f6bbeaa
37522025-04-30T06:42:49.793ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "796e82c9-1f01-4818-b251-ea30015c4b85", "content-length": "702", "date": "Wed, 30 Apr 2025 06:42:48 GMT"} })
37532025-04-30T06:42:49.793ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/0/component/rot/cfpa
37542025-04-30T06:42:49.794ZDEBGtest_snapshot_basic (oximeter-agent): registered new metric producer address = [::1]:43295 collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
37552025-04-30T06:42:49.883ZTRACtest_snapshot_basic (collection-task): top of inner collection loop, waiting for next request collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
37562025-04-30T06:42:49.884ZINFOtest_snapshot_basic (oximeter-agent): refreshed list of producers from Nexus collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 file = oximeter/collector/src/agent.rs:637 n_current_tasks = 2 n_pruned_tasks = 0
37572025-04-30T06:42:49.884ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "d9e58844-dec0-47ff-8a63-bae4770aa46b", "content-length": "727", "date": "Wed, 30 Apr 2025 06:42:48 GMT"} })
37582025-04-30T06:42:49.885ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/0/component/rot/cfpa
37592025-04-30T06:42:49.886ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "eb2c5519-f9c8-4c83-bf53-ee1b6e559e39", "content-length": "729", "date": "Wed, 30 Apr 2025 06:42:48 GMT"} })
37602025-04-30T06:42:49.887ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/0/component/rot/cfpa
37612025-04-30T06:42:49.888ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "bb5a5f14-2267-4ded-9d51-0106da50c291", "content-length": "728", "date": "Wed, 30 Apr 2025 06:42:48 GMT"} })
37622025-04-30T06:42:49.888ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/1
37632025-04-30T06:42:49.892ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "b3bbee26-7480-4ab9-9915-177e1cbe4b44", "content-length": "723", "date": "Wed, 30 Apr 2025 06:42:48 GMT"} })
37642025-04-30T06:42:49.893ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/1/component/sp/caboose?firmware_slot=0
37652025-04-30T06:42:49.902ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "8f8dd959-2372-4606-b446-7d19ced53a3e", "content-length": "109", "date": "Wed, 30 Apr 2025 06:42:48 GMT"} })
37662025-04-30T06:42:49.902ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/1/component/sp/caboose?firmware_slot=1
37672025-04-30T06:42:49.907ZDEBGtest_snapshot_basic: attempt to fetch latest inventory collection result = Ok("found")
37682025-04-30T06:42:49.911ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "8b50f7ec-d6cf-4221-8d07-599d6359e720", "content-length": "109", "date": "Wed, 30 Apr 2025 06:42:48 GMT"} })
37692025-04-30T06:42:49.911ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/1/component/rot/caboose?firmware_slot=0
37702025-04-30T06:42:49.915ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "7ac8ebcf-f820-4926-949f-9dad16080f20", "content-length": "169", "date": "Wed, 30 Apr 2025 06:42:48 GMT"} })
37712025-04-30T06:42:49.915ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/1/component/rot/caboose?firmware_slot=1
37722025-04-30T06:42:49.918ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "82cc8473-f59f-474f-a9ff-6908d2510338", "content-length": "169", "date": "Wed, 30 Apr 2025 06:42:48 GMT"} })
37732025-04-30T06:42:49.918ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/1/component/stage0/caboose?firmware_slot=0
37742025-04-30T06:42:49.921ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "ae305d32-9470-40b1-9c41-97a074171751", "content-length": "178", "date": "Wed, 30 Apr 2025 06:42:48 GMT"} })
37752025-04-30T06:42:49.921ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/1/component/stage0/caboose?firmware_slot=1
37762025-04-30T06:42:49.925ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "7068140c-6cd1-4ca3-9442-3d1c16ae1da2", "content-length": "178", "date": "Wed, 30 Apr 2025 06:42:48 GMT"} })
37772025-04-30T06:42:49.925ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/1/component/rot/cmpa
37782025-04-30T06:42:49.926ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "01f0e546-9292-49f0-adf6-bbd0e27816d8", "content-length": "702", "date": "Wed, 30 Apr 2025 06:42:48 GMT"} })
37792025-04-30T06:42:49.926ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/1/component/rot/cfpa
37802025-04-30T06:42:49.928ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "37623af7-f6c9-4a98-a01d-1367326c8eb0", "content-length": "727", "date": "Wed, 30 Apr 2025 06:42:48 GMT"} })
37812025-04-30T06:42:49.928ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/1/component/rot/cfpa
37822025-04-30T06:42:49.929ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "02210e5f-f909-4601-bc49-a73b7f7f6b1e", "content-length": "729", "date": "Wed, 30 Apr 2025 06:42:48 GMT"} })
37832025-04-30T06:42:49.929ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/1/component/rot/cfpa
37842025-04-30T06:42:49.931ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "1cf04319-6bb1-4e69-b663-c60b73658660", "content-length": "728", "date": "Wed, 30 Apr 2025 06:42:48 GMT"} })
37852025-04-30T06:42:49.931ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
37862025-04-30T06:42:49.931ZTRACtest_snapshot_basic (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
37872025-04-30T06:42:49.931ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
37882025-04-30T06:42:49.931ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
37892025-04-30T06:42:49.932ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
37902025-04-30T06:42:49.935ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
37912025-04-30T06:42:49.945ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
37922025-04-30T06:42:49.945ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
37932025-04-30T06:42:49.945ZTRACtest_snapshot_basic (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
37942025-04-30T06:42:49.945ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
37952025-04-30T06:42:49.945ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
37962025-04-30T06:42:49.947ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
37972025-04-30T06:42:49.949ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
37982025-04-30T06:42:49.954ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
37992025-04-30T06:42:49.954ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
38002025-04-30T06:42:49.954ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
38012025-04-30T06:42:49.955ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
38022025-04-30T06:42:49.959ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
38032025-04-30T06:42:49.959ZTRACtest_snapshot_basic (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
38042025-04-30T06:42:49.959ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
38052025-04-30T06:42:49.959ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {} }
38062025-04-30T06:42:49.961ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
38072025-04-30T06:42:50.094ZDEBGtest_snapshot_basic (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:41923
38082025-04-30T06:42:50.094ZDEBGtest_snapshot_basic (BackgroundTasks): client request SledAgent = http://[::1]:41923 background_task = inventory_collection body = None method = GET uri = http://[::1]:41923/inventory
38092025-04-30T06:42:50.100ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
38102025-04-30T06:42:50.141ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
38112025-04-30T06:42:50.141ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
38122025-04-30T06:42:50.141ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {} }
38132025-04-30T06:42:50.150ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
38142025-04-30T06:42:50.150ZINFOtest_snapshot_basic (dropshot): accepted connection local_addr = [::1]:41923 remote_addr = [::1]:57943 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
38152025-04-30T06:42:50.151ZTRACtest_snapshot_basic (results-sink): batch interval expired, but no samples to insert collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
38162025-04-30T06:42:50.151ZTRACtest_snapshot_basic (results-sink-cluster): batch interval expired, but no samples to insert collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
38172025-04-30T06:42:50.161ZTRACtest_snapshot_basic (dropshot): incoming request local_addr = [::1]:41923 method = GET remote_addr = [::1]:57943 req_id = c332de87-e967-4d14-9ebc-b93e16e716be sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
38182025-04-30T06:42:50.161ZINFOtest_snapshot_basic (dropshot): request completed latency_us = 510 local_addr = [::1]:41923 method = GET remote_addr = [::1]:57943 req_id = c332de87-e967-4d14-9ebc-b93e16e716be response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
38192025-04-30T06:42:50.162ZDEBGtest_snapshot_basic (BackgroundTasks): client response SledAgent = http://[::1]:41923 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:41923/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "c332de87-e967-4d14-9ebc-b93e16e716be", "content-length": "3141", "date": "Wed, 30 Apr 2025 06:42:50 GMT"} })
38202025-04-30T06:42:50.164ZDEBGtest_snapshot_basic (BackgroundTasks): begin collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
38212025-04-30T06:42:50.164ZDEBGtest_snapshot_basic (BackgroundTasks): end collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
38222025-04-30T06:42:50.165ZDEBGtest_snapshot_basic (BackgroundTasks): finished collection background_task = inventory_collection
38232025-04-30T06:42:50.167ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
38242025-04-30T06:42:50.167ZTRACtest_snapshot_basic (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
38252025-04-30T06:42:50.167ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
38262025-04-30T06:42:50.167ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
38272025-04-30T06:42:50.168ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
38282025-04-30T06:42:50.169ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
38292025-04-30T06:42:50.169ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {} }
38302025-04-30T06:42:50.170ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
38312025-04-30T06:42:50.172ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
38322025-04-30T06:42:50.193ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
38332025-04-30T06:42:50.194ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
38342025-04-30T06:42:50.194ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
38352025-04-30T06:42:50.194ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
38362025-04-30T06:42:50.282ZWARNtest_snapshot_basic (datastore): Non-retryable transaction failure err = DatabaseError(UniqueViolation, "duplicate key value violates unique constraint \\"inv_caboose_pkey\\"") transaction = inventory_insert_collection
38372025-04-30T06:42:50.282ZWARNtest_snapshot_basic (BackgroundTasks): inventory collection failed background_task = inventory_collection error = failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint "inv_caboose_pkey"
38382025-04-30T06:42:50.282ZDEBGtest_snapshot_basic (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 788.561367ms iteration = 6 status = {"error":"failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint \\"inv_caboose_pkey\\""}
38392025-04-30T06:42:50.282ZDEBGtest_snapshot_basic (BackgroundTasks): activating background_task = inventory_collection iteration = 7 reason = Signaled
38402025-04-30T06:42:50.282ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
38412025-04-30T06:42:50.283ZTRACtest_snapshot_basic (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
38422025-04-30T06:42:50.283ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
38432025-04-30T06:42:50.283ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
38442025-04-30T06:42:50.284ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
38452025-04-30T06:42:50.290ZDEBGtest_snapshot_basic: attempt to fetch latest inventory collection result = Ok("found")
38462025-04-30T06:42:50.291ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
38472025-04-30T06:42:50.334ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
38482025-04-30T06:42:50.334ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
38492025-04-30T06:42:50.334ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
38502025-04-30T06:42:50.335ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
38512025-04-30T06:42:50.339ZDEBGtest_snapshot_basic (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = [(d1ad44e5-18dd-4193-a70e-bf9042f0e0e1, 0)]
38522025-04-30T06:42:50.339ZTRACtest_snapshot_basic (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
38532025-04-30T06:42:50.340ZTRACtest_snapshot_basic (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: 5e a7 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 ^............_mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:61904 req_id = a124fca1-3186-4fca-96d1-2b61552ecb26
38542025-04-30T06:42:50.340ZDEBGtest_snapshot_basic (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 24231,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:61904 req_id = a124fca1-3186-4fca-96d1-2b61552ecb26
38552025-04-30T06:42:50.340ZDEBGtest_snapshot_basic (store): query key key = _mgs._tcp
38562025-04-30T06:42:50.340ZDEBGtest_snapshot_basic (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
38572025-04-30T06:42:50.341ZDEBGtest_snapshot_basic (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:61904 query = LowerQuery { name: LowerName(Name("_mgs._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 41502, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = a124fca1-3186-4fca-96d1-2b61552ecb26
38582025-04-30T06:42:50.342ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
38592025-04-30T06:42:50.342ZTRACtest_snapshot_basic (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
38602025-04-30T06:42:50.343ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
38612025-04-30T06:42:50.343ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {} }
38622025-04-30T06:42:50.343ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
38632025-04-30T06:42:50.344ZDEBGtest_snapshot_basic (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 41502, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 4235, tv_nsec: 143128178 } })
38642025-04-30T06:42:50.344ZTRACtest_snapshot_basic (DnsResolver): lookup_service_targets: looking up SRV target
38652025-04-30T06:42:50.356ZTRACtest_snapshot_basic (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: e0 bc 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e ............-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:55213 req_id = 09b67da6-4d21-43fd-832b-4eb7bf7aab35
38662025-04-30T06:42:50.356ZDEBGtest_snapshot_basic (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 57532,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:55213 req_id = 09b67da6-4d21-43fd-832b-4eb7bf7aab35
38672025-04-30T06:42:50.357ZDEBGtest_snapshot_basic (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
38682025-04-30T06:42:50.357ZDEBGtest_snapshot_basic (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:55213 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = 09b67da6-4d21-43fd-832b-4eb7bf7aab35
38692025-04-30T06:42:50.418ZTRACtest_snapshot_basic (DnsResolver): lookup_all_socket_v6 srv dns_name = _clickhouse-admin-keeper._tcp.control-plane.oxide.internal
38702025-04-30T06:42:50.418ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
38712025-04-30T06:42:50.430ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
38722025-04-30T06:42:50.430ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
38732025-04-30T06:42:50.430ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {} }
38742025-04-30T06:42:50.431ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
38752025-04-30T06:42:50.431ZTRACtest_snapshot_basic (dns): buffer buffer = Length: 87 (0x57) bytes\n0000: da ff 01 00 00 01 00 00 00 00 00 01 18 5f 63 6c ............._cl\n0010: 69 63 6b 68 6f 75 73 65 2d 61 64 6d 69 6e 2d 6b ickhouse-admin-k\n0020: 65 65 70 65 72 04 5f 74 63 70 0d 63 6f 6e 74 72 eeper._tcp.contr\n0030: 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 69 ol-plane.oxide.i\n0040: 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 04 nternal..!....).\n0050: d0 00 00 00 00 00 00 ....... kind = dns peer_addr = [::1]:51154 req_id = e1ad5efa-76c1-4e58-a807-b138f0b9cdfa
38762025-04-30T06:42:50.431ZDEBGtest_snapshot_basic (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 56063,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 24,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 97,\n 100,\n 109,\n 105,\n 110,\n 45,\n 107,\n 101,\n 101,\n 112,\n 101,\n 114,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:51154 req_id = e1ad5efa-76c1-4e58-a807-b138f0b9cdfa
38772025-04-30T06:42:50.431ZDEBGtest_snapshot_basic (store): query key key = _clickhouse-admin-keeper._tcp
38782025-04-30T06:42:50.432ZERROtest_snapshot_basic (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 56063, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } NXDOMAIN: no records found for name: "_clickhouse-admin-keeper._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:51154 req_id = e1ad5efa-76c1-4e58-a807-b138f0b9cdfa
38792025-04-30T06:42:50.432ZDEBGtest_snapshot_basic (BackgroundTasks): begin collection background_task = inventory_collection
38802025-04-30T06:42:50.432ZDEBGtest_snapshot_basic (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:41502
38812025-04-30T06:42:50.432ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/ignition
38822025-04-30T06:42:50.433ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
38832025-04-30T06:42:50.434ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {} }
38842025-04-30T06:42:50.435ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
38852025-04-30T06:42:50.436ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "47cf1db1-f666-4843-986e-ba88ca041790", "content-length": "835", "date": "Wed, 30 Apr 2025 06:42:50 GMT"} })
38862025-04-30T06:42:50.436ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/0
38872025-04-30T06:42:50.438ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "ceeb71d4-809e-41aa-bbbe-a01b1c97010d", "content-length": "734", "date": "Wed, 30 Apr 2025 06:42:50 GMT"} })
38882025-04-30T06:42:50.438ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/0/component/sp/caboose?firmware_slot=0
38892025-04-30T06:42:50.441ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "2ca674d1-2ba5-45d6-8abd-b134a1603a6d", "content-length": "111", "date": "Wed, 30 Apr 2025 06:42:50 GMT"} })
38902025-04-30T06:42:50.441ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/0/component/sp/caboose?firmware_slot=1
38912025-04-30T06:42:50.458ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "41361bd2-265b-4566-9c2c-62f222cc3a0f", "content-length": "111", "date": "Wed, 30 Apr 2025 06:42:50 GMT"} })
38922025-04-30T06:42:50.458ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/0/component/rot/caboose?firmware_slot=0
38932025-04-30T06:42:50.477ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "c4860124-52c5-4855-b6d1-ce0dd7cdafa7", "content-length": "170", "date": "Wed, 30 Apr 2025 06:42:50 GMT"} })
38942025-04-30T06:42:50.480ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/0/component/rot/caboose?firmware_slot=1
38952025-04-30T06:42:50.484ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "53ad3c66-997c-4d36-bb09-07e644b3a75b", "content-length": "170", "date": "Wed, 30 Apr 2025 06:42:50 GMT"} })
38962025-04-30T06:42:50.484ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/0/component/stage0/caboose?firmware_slot=0
38972025-04-30T06:42:50.487ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "4341fb4c-21a2-4a95-a399-9daa99199cf9", "content-length": "179", "date": "Wed, 30 Apr 2025 06:42:50 GMT"} })
38982025-04-30T06:42:50.489ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/0/component/stage0/caboose?firmware_slot=1
38992025-04-30T06:42:50.495ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "c4402bb3-d694-4c33-9ce6-252eac90c075", "content-length": "179", "date": "Wed, 30 Apr 2025 06:42:50 GMT"} })
39002025-04-30T06:42:50.496ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/0/component/rot/cmpa
39012025-04-30T06:42:50.496ZDEBGtest_snapshot_basic: attempt to fetch latest inventory collection result = Ok("found")
39022025-04-30T06:42:50.498ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "be322300-e614-449d-b211-994e2073b0ac", "content-length": "702", "date": "Wed, 30 Apr 2025 06:42:50 GMT"} })
39032025-04-30T06:42:50.498ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/0/component/rot/cfpa
39042025-04-30T06:42:50.500ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "1073c5c2-741d-45b0-b2de-0522bf5ec2ee", "content-length": "727", "date": "Wed, 30 Apr 2025 06:42:50 GMT"} })
39052025-04-30T06:42:50.500ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/0/component/rot/cfpa
39062025-04-30T06:42:50.502ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "98b7b88a-f801-41a6-85eb-960f2e1347d7", "content-length": "729", "date": "Wed, 30 Apr 2025 06:42:50 GMT"} })
39072025-04-30T06:42:50.502ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/0/component/rot/cfpa
39082025-04-30T06:42:50.503ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "0e51e268-2ce4-4577-94c0-74a42662d58a", "content-length": "728", "date": "Wed, 30 Apr 2025 06:42:50 GMT"} })
39092025-04-30T06:42:50.503ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/1
39102025-04-30T06:42:50.518ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "99e33b99-641b-4bfc-a0ee-0061c14f05a4", "content-length": "734", "date": "Wed, 30 Apr 2025 06:42:50 GMT"} })
39112025-04-30T06:42:50.518ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/1/component/sp/caboose?firmware_slot=0
39122025-04-30T06:42:50.551ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
39132025-04-30T06:42:50.551ZTRACtest_snapshot_basic (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
39142025-04-30T06:42:50.551ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
39152025-04-30T06:42:50.551ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {} }
39162025-04-30T06:42:50.563ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
39172025-04-30T06:42:50.577ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "3b663254-203c-4113-a5b0-025d1ab147e0", "content-length": "111", "date": "Wed, 30 Apr 2025 06:42:50 GMT"} })
39182025-04-30T06:42:50.583ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/1/component/sp/caboose?firmware_slot=1
39192025-04-30T06:42:50.584ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
39202025-04-30T06:42:50.592ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
39212025-04-30T06:42:50.592ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
39222025-04-30T06:42:50.592ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {} }
39232025-04-30T06:42:50.593ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
39242025-04-30T06:42:50.598ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "b6238214-1cb5-46c5-9542-4a9dd15692b0", "content-length": "111", "date": "Wed, 30 Apr 2025 06:42:50 GMT"} })
39252025-04-30T06:42:50.598ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/1/component/rot/caboose?firmware_slot=0
39262025-04-30T06:42:50.599ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
39272025-04-30T06:42:50.599ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {} }
39282025-04-30T06:42:50.599ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
39292025-04-30T06:42:50.607ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "6ec8b8a8-b088-49e7-8370-cbcf99a62302", "content-length": "170", "date": "Wed, 30 Apr 2025 06:42:50 GMT"} })
39302025-04-30T06:42:50.607ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/1/component/rot/caboose?firmware_slot=1
39312025-04-30T06:42:50.611ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "56b9b433-7d2f-4382-8e6f-1b574277ef00", "content-length": "170", "date": "Wed, 30 Apr 2025 06:42:50 GMT"} })
39322025-04-30T06:42:50.612ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/1/component/stage0/caboose?firmware_slot=0
39332025-04-30T06:42:50.618ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "282693fd-efa5-47d1-9110-cdb57e8968f9", "content-length": "179", "date": "Wed, 30 Apr 2025 06:42:50 GMT"} })
39342025-04-30T06:42:50.631ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/1/component/stage0/caboose?firmware_slot=1
39352025-04-30T06:42:50.636ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "e465b438-441c-41d9-bb34-3d7a4c0cb316", "content-length": "179", "date": "Wed, 30 Apr 2025 06:42:50 GMT"} })
39362025-04-30T06:42:50.637ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/1/component/rot/cmpa
39372025-04-30T06:42:50.639ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "fe943cc8-4127-4a6f-872c-ed5ff856a44f", "content-length": "702", "date": "Wed, 30 Apr 2025 06:42:50 GMT"} })
39382025-04-30T06:42:50.639ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/1/component/rot/cfpa
39392025-04-30T06:42:50.641ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "67bc6e06-87fb-4996-adcb-d9ec14a8cbd7", "content-length": "727", "date": "Wed, 30 Apr 2025 06:42:50 GMT"} })
39402025-04-30T06:42:50.641ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/1/component/rot/cfpa
39412025-04-30T06:42:50.644ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "54a7d5ed-09e3-4781-8818-ed3ca3a419bc", "content-length": "729", "date": "Wed, 30 Apr 2025 06:42:50 GMT"} })
39422025-04-30T06:42:50.644ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/1/component/rot/cfpa
39432025-04-30T06:42:50.646ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "08da4961-6541-483c-a0e9-3008f02dba14", "content-length": "728", "date": "Wed, 30 Apr 2025 06:42:50 GMT"} })
39442025-04-30T06:42:50.646ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/0
39452025-04-30T06:42:50.650ZDEBGtest_snapshot_basic: attempt to fetch latest inventory collection result = Ok("found")
39462025-04-30T06:42:50.652ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "7db5ecea-af93-4547-a96b-823a3d51d721", "content-length": "723", "date": "Wed, 30 Apr 2025 06:42:50 GMT"} })
39472025-04-30T06:42:50.653ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/0/component/sp/caboose?firmware_slot=0
39482025-04-30T06:42:50.654ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "ab12be25-0aa5-405f-9619-8c10f6886688", "content-length": "109", "date": "Wed, 30 Apr 2025 06:42:50 GMT"} })
39492025-04-30T06:42:50.654ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/0/component/sp/caboose?firmware_slot=1
39502025-04-30T06:42:50.656ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "61d02af7-e787-4f92-9d49-d564e47f5363", "content-length": "109", "date": "Wed, 30 Apr 2025 06:42:50 GMT"} })
39512025-04-30T06:42:50.657ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/0/component/rot/caboose?firmware_slot=0
39522025-04-30T06:42:50.665ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "1aa4de9a-059a-47cc-b55e-77fe96a604be", "content-length": "169", "date": "Wed, 30 Apr 2025 06:42:50 GMT"} })
39532025-04-30T06:42:50.665ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/0/component/rot/caboose?firmware_slot=1
39542025-04-30T06:42:50.678ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "51ee464f-cc71-4831-b0ff-63e72ddc26ef", "content-length": "169", "date": "Wed, 30 Apr 2025 06:42:50 GMT"} })
39552025-04-30T06:42:50.679ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/0/component/stage0/caboose?firmware_slot=0
39562025-04-30T06:42:50.685ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "7562626e-e80c-4851-a2cf-515bf57c54c4", "content-length": "178", "date": "Wed, 30 Apr 2025 06:42:50 GMT"} })
39572025-04-30T06:42:50.686ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/0/component/stage0/caboose?firmware_slot=1
39582025-04-30T06:42:50.694ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "71f73bb6-fa8d-457b-8914-d2d91c083cc3", "content-length": "178", "date": "Wed, 30 Apr 2025 06:42:50 GMT"} })
39592025-04-30T06:42:50.696ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/0/component/rot/cmpa
39602025-04-30T06:42:50.708ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
39612025-04-30T06:42:50.709ZTRACtest_snapshot_basic (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
39622025-04-30T06:42:50.709ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
39632025-04-30T06:42:50.709ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {} }
39642025-04-30T06:42:50.713ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
39652025-04-30T06:42:50.714ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "ddfd8d45-be9f-484a-9418-191c79d88f9f", "content-length": "702", "date": "Wed, 30 Apr 2025 06:42:50 GMT"} })
39662025-04-30T06:42:50.715ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/0/component/rot/cfpa
39672025-04-30T06:42:50.719ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
39682025-04-30T06:42:50.728ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
39692025-04-30T06:42:50.728ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
39702025-04-30T06:42:50.728ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {} }
39712025-04-30T06:42:50.728ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
39722025-04-30T06:42:50.729ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "870b1061-cd15-46ac-aa51-7ef8f28bbdb1", "content-length": "727", "date": "Wed, 30 Apr 2025 06:42:50 GMT"} })
39732025-04-30T06:42:50.729ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/0/component/rot/cfpa
39742025-04-30T06:42:50.734ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "24d3e0bd-34f2-4706-8ce3-ef0817a503c0", "content-length": "729", "date": "Wed, 30 Apr 2025 06:42:50 GMT"} })
39752025-04-30T06:42:50.735ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/0/component/rot/cfpa
39762025-04-30T06:42:50.735ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
39772025-04-30T06:42:50.735ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {} }
39782025-04-30T06:42:50.736ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
39792025-04-30T06:42:50.738ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "c2e8e81f-bf8e-4c2e-9a27-1a8da93f5523", "content-length": "728", "date": "Wed, 30 Apr 2025 06:42:50 GMT"} })
39802025-04-30T06:42:50.738ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/1
39812025-04-30T06:42:50.742ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "333d214c-9662-476d-b06d-43e9e31e63a0", "content-length": "723", "date": "Wed, 30 Apr 2025 06:42:50 GMT"} })
39822025-04-30T06:42:50.743ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/1/component/sp/caboose?firmware_slot=0
39832025-04-30T06:42:50.748ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "1ab39d41-d4b7-4f61-a85d-0473e3a61876", "content-length": "109", "date": "Wed, 30 Apr 2025 06:42:50 GMT"} })
39842025-04-30T06:42:50.748ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/1/component/sp/caboose?firmware_slot=1
39852025-04-30T06:42:50.768ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "23a00f7f-671e-435c-9124-7f6030f4416d", "content-length": "109", "date": "Wed, 30 Apr 2025 06:42:50 GMT"} })
39862025-04-30T06:42:50.769ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/1/component/rot/caboose?firmware_slot=0
39872025-04-30T06:42:50.779ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "8750decb-0c20-4289-967f-1cb5eadc5ee4", "content-length": "169", "date": "Wed, 30 Apr 2025 06:42:50 GMT"} })
39882025-04-30T06:42:50.779ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/1/component/rot/caboose?firmware_slot=1
39892025-04-30T06:42:50.789ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "ec2a241c-e4fc-452f-95ee-fb41cf7d3310", "content-length": "169", "date": "Wed, 30 Apr 2025 06:42:50 GMT"} })
39902025-04-30T06:42:50.789ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/1/component/stage0/caboose?firmware_slot=0
39912025-04-30T06:42:50.791ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "ee938de2-20b1-4f53-9639-faff0f57f4c3", "content-length": "178", "date": "Wed, 30 Apr 2025 06:42:50 GMT"} })
39922025-04-30T06:42:50.791ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/1/component/stage0/caboose?firmware_slot=1
39932025-04-30T06:42:50.794ZDEBGtest_snapshot_basic (collection-task): sent timer-based collection request to the collection task collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
39942025-04-30T06:42:50.794ZDEBGtest_snapshot_basic (collection-task): collecting from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
39952025-04-30T06:42:50.795ZDEBGtest_snapshot_basic: attempt to fetch latest inventory collection result = Ok("found")
39962025-04-30T06:42:50.800ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "730d32a8-03ab-4a7f-9062-b67cc935182f", "content-length": "178", "date": "Wed, 30 Apr 2025 06:42:50 GMT"} })
39972025-04-30T06:42:50.800ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/1/component/rot/cmpa
39982025-04-30T06:42:50.800ZTRACtest_snapshot_basic (collection-task): sent collection request to producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
39992025-04-30T06:42:50.801ZDEBGtest_snapshot_basic (collection-task): collected results from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 n_results = 1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
40002025-04-30T06:42:50.801ZTRACtest_snapshot_basic (collection-task): forwarded results to main collection loop collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
40012025-04-30T06:42:50.801ZTRACtest_snapshot_basic (collection-task): top of inner collection loop, waiting for next request collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
40022025-04-30T06:42:50.802ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "a6d68b78-76f8-45d5-8bbe-c78085285b55", "content-length": "702", "date": "Wed, 30 Apr 2025 06:42:50 GMT"} })
40032025-04-30T06:42:50.802ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/1/component/rot/cfpa
40042025-04-30T06:42:50.803ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "d38499ca-04bf-45f9-8ea8-67e625b2eacd", "content-length": "727", "date": "Wed, 30 Apr 2025 06:42:50 GMT"} })
40052025-04-30T06:42:50.803ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/1/component/rot/cfpa
40062025-04-30T06:42:50.804ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "f610de31-7604-4f85-b5bd-84e66b80eeee", "content-length": "729", "date": "Wed, 30 Apr 2025 06:42:50 GMT"} })
40072025-04-30T06:42:50.805ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/1/component/rot/cfpa
40082025-04-30T06:42:50.808ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "ca2440c2-ca1c-4102-bd8a-3f482fe65a61", "content-length": "728", "date": "Wed, 30 Apr 2025 06:42:50 GMT"} })
40092025-04-30T06:42:50.808ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
40102025-04-30T06:42:50.808ZTRACtest_snapshot_basic (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
40112025-04-30T06:42:50.808ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
40122025-04-30T06:42:50.808ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
40132025-04-30T06:42:50.810ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
40142025-04-30T06:42:50.827ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
40152025-04-30T06:42:50.839ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
40162025-04-30T06:42:50.839ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
40172025-04-30T06:42:50.839ZTRACtest_snapshot_basic (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
40182025-04-30T06:42:50.839ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
40192025-04-30T06:42:50.839ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
40202025-04-30T06:42:50.840ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
40212025-04-30T06:42:50.844ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
40222025-04-30T06:42:50.851ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
40232025-04-30T06:42:50.851ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
40242025-04-30T06:42:50.851ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
40252025-04-30T06:42:50.852ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
40262025-04-30T06:42:50.853ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
40272025-04-30T06:42:50.853ZTRACtest_snapshot_basic (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
40282025-04-30T06:42:50.853ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
40292025-04-30T06:42:50.853ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {} }
40302025-04-30T06:42:50.854ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
40312025-04-30T06:42:50.939ZDEBGtest_snapshot_basic (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:41923
40322025-04-30T06:42:50.939ZDEBGtest_snapshot_basic (BackgroundTasks): client request SledAgent = http://[::1]:41923 background_task = inventory_collection body = None method = GET uri = http://[::1]:41923/inventory
40332025-04-30T06:42:50.940ZINFOtest_snapshot_basic (dropshot): accepted connection local_addr = [::1]:41923 remote_addr = [::1]:65315 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
40342025-04-30T06:42:50.940ZTRACtest_snapshot_basic (dropshot): incoming request local_addr = [::1]:41923 method = GET remote_addr = [::1]:65315 req_id = 22f73516-326a-4563-88dd-94444731cd3c sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
40352025-04-30T06:42:50.941ZINFOtest_snapshot_basic (dropshot): request completed latency_us = 525 local_addr = [::1]:41923 method = GET remote_addr = [::1]:65315 req_id = 22f73516-326a-4563-88dd-94444731cd3c response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
40362025-04-30T06:42:50.941ZDEBGtest_snapshot_basic (BackgroundTasks): client response SledAgent = http://[::1]:41923 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:41923/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "22f73516-326a-4563-88dd-94444731cd3c", "content-length": "3141", "date": "Wed, 30 Apr 2025 06:42:50 GMT"} })
40372025-04-30T06:42:50.943ZDEBGtest_snapshot_basic (BackgroundTasks): begin collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
40382025-04-30T06:42:50.943ZDEBGtest_snapshot_basic (BackgroundTasks): end collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
40392025-04-30T06:42:50.943ZDEBGtest_snapshot_basic (BackgroundTasks): finished collection background_task = inventory_collection
40402025-04-30T06:42:50.945ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
40412025-04-30T06:42:50.945ZTRACtest_snapshot_basic (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
40422025-04-30T06:42:50.945ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
40432025-04-30T06:42:50.945ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
40442025-04-30T06:42:50.946ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
40452025-04-30T06:42:50.946ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
40462025-04-30T06:42:50.953ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
40472025-04-30T06:42:50.953ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
40482025-04-30T06:42:50.953ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {} }
40492025-04-30T06:42:50.954ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
40502025-04-30T06:42:50.956ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
40512025-04-30T06:42:50.960ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
40522025-04-30T06:42:50.961ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
40532025-04-30T06:42:50.961ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
40542025-04-30T06:42:50.962ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
40552025-04-30T06:42:50.965ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
40562025-04-30T06:42:50.965ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {} }
40572025-04-30T06:42:50.966ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
40582025-04-30T06:42:51.028ZWARNtest_snapshot_basic (datastore): Non-retryable transaction failure err = DatabaseError(UniqueViolation, "duplicate key value violates unique constraint \\"inv_caboose_pkey\\"") transaction = inventory_insert_collection
40592025-04-30T06:42:51.028ZWARNtest_snapshot_basic (BackgroundTasks): inventory collection failed background_task = inventory_collection error = failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint "inv_caboose_pkey"
40602025-04-30T06:42:51.029ZDEBGtest_snapshot_basic (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 746.118582ms iteration = 7 status = {"error":"failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint \\"inv_caboose_pkey\\""}
40612025-04-30T06:42:51.029ZDEBGtest_snapshot_basic (BackgroundTasks): activating background_task = inventory_collection iteration = 8 reason = Signaled
40622025-04-30T06:42:51.029ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
40632025-04-30T06:42:51.029ZTRACtest_snapshot_basic (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
40642025-04-30T06:42:51.029ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
40652025-04-30T06:42:51.029ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
40662025-04-30T06:42:51.030ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
40672025-04-30T06:42:51.045ZDEBGtest_snapshot_basic (results-sink): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
40682025-04-30T06:42:51.047ZDEBGtest_snapshot_basic (results-sink-cluster): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
40692025-04-30T06:42:51.047ZDEBGtest_snapshot_basic (clickhouse-client): unrolling 1 total samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 6ad06c6b-9a6e-4ed1-b3f4-5d2e9da3f8a7
40702025-04-30T06:42:51.049ZDEBGtest_snapshot_basic (clickhouse-client): retrieving timeseries schema from database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 6ad06c6b-9a6e-4ed1-b3f4-5d2e9da3f8a7
40712025-04-30T06:42:51.049ZTRACtest_snapshot_basic (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 6ad06c6b-9a6e-4ed1-b3f4-5d2e9da3f8a7 sql = SELECT * FROM oximeter.timeseries_schema FORMAT Native;
40722025-04-30T06:42:51.049ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
40732025-04-30T06:42:51.057ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
40742025-04-30T06:42:51.057ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
40752025-04-30T06:42:51.057ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
40762025-04-30T06:42:51.059ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
40772025-04-30T06:42:51.059ZDEBGtest_snapshot_basic: attempt to fetch latest inventory collection result = Ok("found")
40782025-04-30T06:42:51.061ZTRACtest_snapshot_basic (clickhouse-client): no new timeseries schema in database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 6ad06c6b-9a6e-4ed1-b3f4-5d2e9da3f8a7
40792025-04-30T06:42:51.061ZDEBGtest_snapshot_basic (clickhouse-client): new timeseries schema collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 6ad06c6b-9a6e-4ed1-b3f4-5d2e9da3f8a7 schema = TimeseriesSchema { timeseries_name: TimeseriesName("integration_target:integration_metric"), description: TimeseriesDescription { target: "", metric: "" }, field_schema: {FieldSchema { name: "metric_name", field_type: String, source: Metric, description: "" }, FieldSchema { name: "target_name", field_type: String, source: Target, description: "" }}, datum_type: I64, version: 1, authz_scope: Fleet, units: Count, created: 2025-04-30T06:42:51.048988470Z } timeseries_name = integration_target:integration_metric
40802025-04-30T06:42:51.061ZDEBGtest_snapshot_basic (clickhouse-client): inserting 1 new timeseries schema collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 6ad06c6b-9a6e-4ed1-b3f4-5d2e9da3f8a7
40812025-04-30T06:42:51.062ZTRACtest_snapshot_basic (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 6ad06c6b-9a6e-4ed1-b3f4-5d2e9da3f8a7 n_columns = 6 n_rows = 1 sql = INSERT INTO oximeter.timeseries_schema FORMAT Native
40822025-04-30T06:42:51.067ZDEBGtest_snapshot_basic (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = [(d1ad44e5-18dd-4193-a70e-bf9042f0e0e1, 0)]
40832025-04-30T06:42:51.068ZTRACtest_snapshot_basic (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
40842025-04-30T06:42:51.078ZTRACtest_snapshot_basic (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 6ad06c6b-9a6e-4ed1-b3f4-5d2e9da3f8a7 n_columns = 4 n_rows = 2 sql = INSERT INTO oximeter.fields_string FORMAT Native
40852025-04-30T06:42:51.078ZTRACtest_snapshot_basic (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: 57 73 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 Ws..........._mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:58583 req_id = 27602a1a-b047-4dc3-b730-6c7f2b0a3852
40862025-04-30T06:42:51.078ZDEBGtest_snapshot_basic (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 22387,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:58583 req_id = 27602a1a-b047-4dc3-b730-6c7f2b0a3852
40872025-04-30T06:42:51.078ZDEBGtest_snapshot_basic (store): query key key = _mgs._tcp
40882025-04-30T06:42:51.079ZDEBGtest_snapshot_basic (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
40892025-04-30T06:42:51.079ZDEBGtest_snapshot_basic (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:58583 query = LowerQuery { name: LowerName(Name("_mgs._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 41502, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = 27602a1a-b047-4dc3-b730-6c7f2b0a3852
40902025-04-30T06:42:51.080ZDEBGtest_snapshot_basic (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 41502, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 4235, tv_nsec: 879291986 } })
40912025-04-30T06:42:51.080ZTRACtest_snapshot_basic (DnsResolver): lookup_service_targets: looking up SRV target
40922025-04-30T06:42:51.081ZTRACtest_snapshot_basic (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: 9e b7 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e ............-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:51320 req_id = cae993e9-06e2-4bbd-95eb-2ba5e4db3b94
40932025-04-30T06:42:51.081ZDEBGtest_snapshot_basic (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 40631,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:51320 req_id = cae993e9-06e2-4bbd-95eb-2ba5e4db3b94
40942025-04-30T06:42:51.081ZDEBGtest_snapshot_basic (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
40952025-04-30T06:42:51.081ZDEBGtest_snapshot_basic (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:51320 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = cae993e9-06e2-4bbd-95eb-2ba5e4db3b94
40962025-04-30T06:42:51.154ZTRACtest_snapshot_basic (DnsResolver): lookup_all_socket_v6 srv dns_name = _clickhouse-admin-keeper._tcp.control-plane.oxide.internal
40972025-04-30T06:42:51.154ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
40982025-04-30T06:42:51.154ZTRACtest_snapshot_basic (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
40992025-04-30T06:42:51.154ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
41002025-04-30T06:42:51.154ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {} }
41012025-04-30T06:42:51.156ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
41022025-04-30T06:42:51.156ZDEBGtest_snapshot_basic (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 6ad06c6b-9a6e-4ed1-b3f4-5d2e9da3f8a7 n_rows = 2 table_name = fields_string
41032025-04-30T06:42:51.156ZTRACtest_snapshot_basic (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 6ad06c6b-9a6e-4ed1-b3f4-5d2e9da3f8a7 n_columns = 4 n_rows = 1 sql = INSERT INTO oximeter.measurements_i64 FORMAT Native
41042025-04-30T06:42:51.158ZWARNtest_snapshot_basic (results-sink-cluster): failed to insert some results into metric DB collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 error = Failed to check out connection to database: No backends found for this service file = oximeter/collector/src/results_sink.rs:92
41052025-04-30T06:42:51.158ZTRACtest_snapshot_basic (dns): buffer buffer = Length: 87 (0x57) bytes\n0000: 7b 78 01 00 00 01 00 00 00 00 00 01 18 5f 63 6c {x..........._cl\n0010: 69 63 6b 68 6f 75 73 65 2d 61 64 6d 69 6e 2d 6b ickhouse-admin-k\n0020: 65 65 70 65 72 04 5f 74 63 70 0d 63 6f 6e 74 72 eeper._tcp.contr\n0030: 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 69 ol-plane.oxide.i\n0040: 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 04 nternal..!....).\n0050: d0 00 00 00 00 00 00 ....... kind = dns peer_addr = [::1]:52743 req_id = 02535d6c-471a-4ed3-9b8b-4040576ab6f5
41062025-04-30T06:42:51.158ZDEBGtest_snapshot_basic (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 31608,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 24,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 97,\n 100,\n 109,\n 105,\n 110,\n 45,\n 107,\n 101,\n 101,\n 112,\n 101,\n 114,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:52743 req_id = 02535d6c-471a-4ed3-9b8b-4040576ab6f5
41072025-04-30T06:42:51.159ZDEBGtest_snapshot_basic (store): query key key = _clickhouse-admin-keeper._tcp
41082025-04-30T06:42:51.159ZERROtest_snapshot_basic (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 31608, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } NXDOMAIN: no records found for name: "_clickhouse-admin-keeper._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:52743 req_id = 02535d6c-471a-4ed3-9b8b-4040576ab6f5
41092025-04-30T06:42:51.159ZDEBGtest_snapshot_basic (BackgroundTasks): begin collection background_task = inventory_collection
41102025-04-30T06:42:51.159ZDEBGtest_snapshot_basic (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:41502
41112025-04-30T06:42:51.159ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/ignition
41122025-04-30T06:42:51.160ZDEBGtest_snapshot_basic (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 6ad06c6b-9a6e-4ed1-b3f4-5d2e9da3f8a7 n_rows = 1 table_name = measurements_i64
41132025-04-30T06:42:51.160ZTRACtest_snapshot_basic (results-sink): successfully inserted samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
41142025-04-30T06:42:51.161ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
41152025-04-30T06:42:51.167ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
41162025-04-30T06:42:51.167ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
41172025-04-30T06:42:51.167ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {} }
41182025-04-30T06:42:51.169ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
41192025-04-30T06:42:51.169ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "78b8509c-ebe9-4011-9985-278bd85de8fa", "content-length": "835", "date": "Wed, 30 Apr 2025 06:42:51 GMT"} })
41202025-04-30T06:42:51.172ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/0
41212025-04-30T06:42:51.175ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
41222025-04-30T06:42:51.175ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {} }
41232025-04-30T06:42:51.176ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
41242025-04-30T06:42:51.178ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "32efcb94-5aff-45e5-b907-f633ddef4106", "content-length": "734", "date": "Wed, 30 Apr 2025 06:42:51 GMT"} })
41252025-04-30T06:42:51.178ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/0/component/sp/caboose?firmware_slot=0
41262025-04-30T06:42:51.181ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "48b0beff-039d-47ae-95d2-62e0160f7496", "content-length": "111", "date": "Wed, 30 Apr 2025 06:42:51 GMT"} })
41272025-04-30T06:42:51.182ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/0/component/sp/caboose?firmware_slot=1
41282025-04-30T06:42:51.184ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "41963398-8603-448e-936b-613519458629", "content-length": "111", "date": "Wed, 30 Apr 2025 06:42:51 GMT"} })
41292025-04-30T06:42:51.184ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/0/component/rot/caboose?firmware_slot=0
41302025-04-30T06:42:51.190ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "9f4cc7ba-b47c-4239-9de0-8693f65d4f00", "content-length": "170", "date": "Wed, 30 Apr 2025 06:42:51 GMT"} })
41312025-04-30T06:42:51.190ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/0/component/rot/caboose?firmware_slot=1
41322025-04-30T06:42:51.228ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "ad5e9952-ef9e-4048-9cca-b4bd7638b4b7", "content-length": "170", "date": "Wed, 30 Apr 2025 06:42:51 GMT"} })
41332025-04-30T06:42:51.228ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/0/component/stage0/caboose?firmware_slot=0
41342025-04-30T06:42:51.230ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "128a5a0f-042f-4b54-9c06-a495669ec090", "content-length": "179", "date": "Wed, 30 Apr 2025 06:42:51 GMT"} })
41352025-04-30T06:42:51.230ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/0/component/stage0/caboose?firmware_slot=1
41362025-04-30T06:42:51.232ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "af83ecb8-93b7-497a-b11c-2816bc83e868", "content-length": "179", "date": "Wed, 30 Apr 2025 06:42:51 GMT"} })
41372025-04-30T06:42:51.232ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/0/component/rot/cmpa
41382025-04-30T06:42:51.233ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "c9f683ad-b032-4b54-8f18-4543ff0fa122", "content-length": "702", "date": "Wed, 30 Apr 2025 06:42:51 GMT"} })
41392025-04-30T06:42:51.233ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/0/component/rot/cfpa
41402025-04-30T06:42:51.234ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "1401f356-4283-4cea-b31d-d6ebdc1d10a4", "content-length": "727", "date": "Wed, 30 Apr 2025 06:42:51 GMT"} })
41412025-04-30T06:42:51.234ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/0/component/rot/cfpa
41422025-04-30T06:42:51.235ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "6564f047-45b3-40f6-b84f-c31a6719f01f", "content-length": "729", "date": "Wed, 30 Apr 2025 06:42:51 GMT"} })
41432025-04-30T06:42:51.235ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/0/component/rot/cfpa
41442025-04-30T06:42:51.249ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "452da6f7-0206-4333-9cb4-63f1470c6585", "content-length": "728", "date": "Wed, 30 Apr 2025 06:42:51 GMT"} })
41452025-04-30T06:42:51.249ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/1
41462025-04-30T06:42:51.250ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "1cf21e5e-9b33-495e-9857-485c06f3ce90", "content-length": "734", "date": "Wed, 30 Apr 2025 06:42:51 GMT"} })
41472025-04-30T06:42:51.250ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/1/component/sp/caboose?firmware_slot=0
41482025-04-30T06:42:51.253ZDEBGtest_snapshot_basic: attempt to fetch latest inventory collection result = Ok("found")
41492025-04-30T06:42:51.265ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "7750eceb-6b79-4f11-9c74-95e5f18b07ba", "content-length": "111", "date": "Wed, 30 Apr 2025 06:42:51 GMT"} })
41502025-04-30T06:42:51.265ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/1/component/sp/caboose?firmware_slot=1
41512025-04-30T06:42:51.267ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "e1e783d4-367f-4358-8b04-8f9bccdc5108", "content-length": "111", "date": "Wed, 30 Apr 2025 06:42:51 GMT"} })
41522025-04-30T06:42:51.267ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/1/component/rot/caboose?firmware_slot=0
41532025-04-30T06:42:51.270ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "3741b71b-054a-4a29-a4bf-61b9d5b6a2bd", "content-length": "170", "date": "Wed, 30 Apr 2025 06:42:51 GMT"} })
41542025-04-30T06:42:51.270ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/1/component/rot/caboose?firmware_slot=1
41552025-04-30T06:42:51.276ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "afdcef95-a92e-451b-bf72-6526d589e533", "content-length": "170", "date": "Wed, 30 Apr 2025 06:42:51 GMT"} })
41562025-04-30T06:42:51.276ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/1/component/stage0/caboose?firmware_slot=0
41572025-04-30T06:42:51.277ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "2800e228-3b73-47c9-b137-0ac277b4b693", "content-length": "179", "date": "Wed, 30 Apr 2025 06:42:51 GMT"} })
41582025-04-30T06:42:51.277ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/1/component/stage0/caboose?firmware_slot=1
41592025-04-30T06:42:51.279ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "3b13835c-9ac2-4819-bd8d-db9abc2661f9", "content-length": "179", "date": "Wed, 30 Apr 2025 06:42:51 GMT"} })
41602025-04-30T06:42:51.279ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/1/component/rot/cmpa
41612025-04-30T06:42:51.280ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "73ab0944-0a80-4b98-a30d-98f84b670a67", "content-length": "702", "date": "Wed, 30 Apr 2025 06:42:51 GMT"} })
41622025-04-30T06:42:51.281ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/1/component/rot/cfpa
41632025-04-30T06:42:51.282ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "51f3a431-5ae0-4035-a6ed-de0157035094", "content-length": "727", "date": "Wed, 30 Apr 2025 06:42:51 GMT"} })
41642025-04-30T06:42:51.283ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/1/component/rot/cfpa
41652025-04-30T06:42:51.285ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "b73a203b-cbfb-4358-a085-b801ea53fc93", "content-length": "729", "date": "Wed, 30 Apr 2025 06:42:51 GMT"} })
41662025-04-30T06:42:51.286ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/1/component/rot/cfpa
41672025-04-30T06:42:51.286ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "8467398e-24ab-4e33-bbe3-099990ca4525", "content-length": "728", "date": "Wed, 30 Apr 2025 06:42:51 GMT"} })
41682025-04-30T06:42:51.287ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/0
41692025-04-30T06:42:51.297ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "c1db5bc0-f8a5-402a-9736-eed3c2521d95", "content-length": "723", "date": "Wed, 30 Apr 2025 06:42:51 GMT"} })
41702025-04-30T06:42:51.297ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/0/component/sp/caboose?firmware_slot=0
41712025-04-30T06:42:51.307ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
41722025-04-30T06:42:51.307ZTRACtest_snapshot_basic (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
41732025-04-30T06:42:51.307ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
41742025-04-30T06:42:51.307ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {} }
41752025-04-30T06:42:51.310ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
41762025-04-30T06:42:51.312ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "b9feba02-9ec2-4a7c-8083-09279dd7ddce", "content-length": "109", "date": "Wed, 30 Apr 2025 06:42:51 GMT"} })
41772025-04-30T06:42:51.312ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/0/component/sp/caboose?firmware_slot=1
41782025-04-30T06:42:51.314ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
41792025-04-30T06:42:51.320ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
41802025-04-30T06:42:51.320ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
41812025-04-30T06:42:51.320ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {} }
41822025-04-30T06:42:51.320ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
41832025-04-30T06:42:51.327ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
41842025-04-30T06:42:51.327ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {} }
41852025-04-30T06:42:51.328ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
41862025-04-30T06:42:51.329ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "fb313379-622a-4825-9b8b-262876d46855", "content-length": "109", "date": "Wed, 30 Apr 2025 06:42:51 GMT"} })
41872025-04-30T06:42:51.329ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/0/component/rot/caboose?firmware_slot=0
41882025-04-30T06:42:51.334ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "c14f2ddf-b485-40d9-b2a8-2875b4a26939", "content-length": "169", "date": "Wed, 30 Apr 2025 06:42:51 GMT"} })
41892025-04-30T06:42:51.334ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/0/component/rot/caboose?firmware_slot=1
41902025-04-30T06:42:51.339ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "bb8464e3-06e5-4342-ae6a-654696e62c07", "content-length": "169", "date": "Wed, 30 Apr 2025 06:42:51 GMT"} })
41912025-04-30T06:42:51.339ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/0/component/stage0/caboose?firmware_slot=0
41922025-04-30T06:42:51.341ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "f77af8e6-29c9-47e3-b13d-2f366ea16b69", "content-length": "178", "date": "Wed, 30 Apr 2025 06:42:51 GMT"} })
41932025-04-30T06:42:51.342ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/0/component/stage0/caboose?firmware_slot=1
41942025-04-30T06:42:51.343ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "9052f3ff-e106-4cbe-b709-e453f91ba78a", "content-length": "178", "date": "Wed, 30 Apr 2025 06:42:51 GMT"} })
41952025-04-30T06:42:51.344ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/0/component/rot/cmpa
41962025-04-30T06:42:51.345ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "80eed9bf-f0a5-4381-9450-0d3773bf7b2a", "content-length": "702", "date": "Wed, 30 Apr 2025 06:42:51 GMT"} })
41972025-04-30T06:42:51.349ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/0/component/rot/cfpa
41982025-04-30T06:42:51.357ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "794267c4-8791-4c63-a041-48cef46e51c5", "content-length": "727", "date": "Wed, 30 Apr 2025 06:42:51 GMT"} })
41992025-04-30T06:42:51.357ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/0/component/rot/cfpa
42002025-04-30T06:42:51.361ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "139a1f43-bd79-481e-a369-9299b4c348d8", "content-length": "729", "date": "Wed, 30 Apr 2025 06:42:51 GMT"} })
42012025-04-30T06:42:51.361ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/0/component/rot/cfpa
42022025-04-30T06:42:51.361ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "a90f7fa9-30a7-4a9a-bfcf-a3c160e268c6", "content-length": "728", "date": "Wed, 30 Apr 2025 06:42:51 GMT"} })
42032025-04-30T06:42:51.361ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/1
42042025-04-30T06:42:51.361ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "029fffe6-d39f-4415-ae75-94208ef1fc06", "content-length": "723", "date": "Wed, 30 Apr 2025 06:42:51 GMT"} })
42052025-04-30T06:42:51.361ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/1/component/sp/caboose?firmware_slot=0
42062025-04-30T06:42:51.363ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "24981713-603f-451e-a9fe-49ec683a4371", "content-length": "109", "date": "Wed, 30 Apr 2025 06:42:51 GMT"} })
42072025-04-30T06:42:51.363ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/1/component/sp/caboose?firmware_slot=1
42082025-04-30T06:42:51.371ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "1bdfd6a7-5676-4d15-8950-194ff1ccc695", "content-length": "109", "date": "Wed, 30 Apr 2025 06:42:51 GMT"} })
42092025-04-30T06:42:51.371ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/1/component/rot/caboose?firmware_slot=0
42102025-04-30T06:42:51.373ZDEBGtest_snapshot_basic: attempt to fetch latest inventory collection result = Ok("found")
42112025-04-30T06:42:51.377ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "3f6ddc1d-190e-4933-9158-ba5868b61c11", "content-length": "169", "date": "Wed, 30 Apr 2025 06:42:51 GMT"} })
42122025-04-30T06:42:51.378ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/1/component/rot/caboose?firmware_slot=1
42132025-04-30T06:42:51.380ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "ed2eae0d-296e-45cc-a05f-c029686daefd", "content-length": "169", "date": "Wed, 30 Apr 2025 06:42:51 GMT"} })
42142025-04-30T06:42:51.380ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/1/component/stage0/caboose?firmware_slot=0
42152025-04-30T06:42:51.382ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "bac3c9fa-31f5-4996-962b-e71da15ea891", "content-length": "178", "date": "Wed, 30 Apr 2025 06:42:51 GMT"} })
42162025-04-30T06:42:51.382ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/1/component/stage0/caboose?firmware_slot=1
42172025-04-30T06:42:51.384ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "870d37b2-d18e-410f-a6dc-c26e83c05c97", "content-length": "178", "date": "Wed, 30 Apr 2025 06:42:51 GMT"} })
42182025-04-30T06:42:51.384ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/1/component/rot/cmpa
42192025-04-30T06:42:51.385ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "93fa0c04-3ce2-4b3d-94db-0b6d0d567cd9", "content-length": "702", "date": "Wed, 30 Apr 2025 06:42:51 GMT"} })
42202025-04-30T06:42:51.385ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/1/component/rot/cfpa
42212025-04-30T06:42:51.386ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "ed2e21a0-c325-4252-91a3-2b8dd1a36d1c", "content-length": "727", "date": "Wed, 30 Apr 2025 06:42:51 GMT"} })
42222025-04-30T06:42:51.386ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/1/component/rot/cfpa
42232025-04-30T06:42:51.387ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "5b81094f-945d-4980-baa1-b22c903bdcbe", "content-length": "729", "date": "Wed, 30 Apr 2025 06:42:51 GMT"} })
42242025-04-30T06:42:51.387ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/1/component/rot/cfpa
42252025-04-30T06:42:51.389ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "a52c259d-b20c-4f2b-a6f3-58b2572e37fd", "content-length": "728", "date": "Wed, 30 Apr 2025 06:42:51 GMT"} })
42262025-04-30T06:42:51.389ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
42272025-04-30T06:42:51.389ZTRACtest_snapshot_basic (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
42282025-04-30T06:42:51.389ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
42292025-04-30T06:42:51.389ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
42302025-04-30T06:42:51.390ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
42312025-04-30T06:42:51.409ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
42322025-04-30T06:42:51.415ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
42332025-04-30T06:42:51.415ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
42342025-04-30T06:42:51.415ZTRACtest_snapshot_basic (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
42352025-04-30T06:42:51.415ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
42362025-04-30T06:42:51.426ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
42372025-04-30T06:42:51.426ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
42382025-04-30T06:42:51.432ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
42392025-04-30T06:42:51.432ZTRACtest_snapshot_basic (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
42402025-04-30T06:42:51.432ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
42412025-04-30T06:42:51.432ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {} }
42422025-04-30T06:42:51.432ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
42432025-04-30T06:42:51.433ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
42442025-04-30T06:42:51.436ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
42452025-04-30T06:42:51.436ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
42462025-04-30T06:42:51.436ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
42472025-04-30T06:42:51.436ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
42482025-04-30T06:42:51.438ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
42492025-04-30T06:42:51.445ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
42502025-04-30T06:42:51.445ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
42512025-04-30T06:42:51.445ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {} }
42522025-04-30T06:42:51.446ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
42532025-04-30T06:42:51.520ZDEBGtest_snapshot_basic (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:41923
42542025-04-30T06:42:51.520ZDEBGtest_snapshot_basic (BackgroundTasks): client request SledAgent = http://[::1]:41923 background_task = inventory_collection body = None method = GET uri = http://[::1]:41923/inventory
42552025-04-30T06:42:51.521ZINFOtest_snapshot_basic (dropshot): accepted connection local_addr = [::1]:41923 remote_addr = [::1]:54161 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
42562025-04-30T06:42:51.523ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
42572025-04-30T06:42:51.523ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {} }
42582025-04-30T06:42:51.525ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
42592025-04-30T06:42:51.526ZTRACtest_snapshot_basic (dropshot): incoming request local_addr = [::1]:41923 method = GET remote_addr = [::1]:54161 req_id = 22d09388-4549-4111-a0d3-da45c2bb376c sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
42602025-04-30T06:42:51.528ZINFOtest_snapshot_basic (dropshot): request completed latency_us = 2322 local_addr = [::1]:41923 method = GET remote_addr = [::1]:54161 req_id = 22d09388-4549-4111-a0d3-da45c2bb376c response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
42612025-04-30T06:42:51.532ZDEBGtest_snapshot_basic (BackgroundTasks): client response SledAgent = http://[::1]:41923 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:41923/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "22d09388-4549-4111-a0d3-da45c2bb376c", "content-length": "3141", "date": "Wed, 30 Apr 2025 06:42:51 GMT"} })
42622025-04-30T06:42:51.536ZDEBGtest_snapshot_basic (BackgroundTasks): begin collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
42632025-04-30T06:42:51.536ZDEBGtest_snapshot_basic (BackgroundTasks): end collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
42642025-04-30T06:42:51.536ZDEBGtest_snapshot_basic (BackgroundTasks): finished collection background_task = inventory_collection
42652025-04-30T06:42:51.538ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
42662025-04-30T06:42:51.538ZTRACtest_snapshot_basic (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
42672025-04-30T06:42:51.539ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
42682025-04-30T06:42:51.539ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
42692025-04-30T06:42:51.540ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
42702025-04-30T06:42:51.551ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
42712025-04-30T06:42:51.561ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
42722025-04-30T06:42:51.561ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
42732025-04-30T06:42:51.561ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
42742025-04-30T06:42:51.562ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
42752025-04-30T06:42:51.612ZDEBGtest_snapshot_basic: attempt to fetch latest inventory collection result = Ok("found")
42762025-04-30T06:42:51.634ZWARNtest_snapshot_basic (datastore): Non-retryable transaction failure err = DatabaseError(UniqueViolation, "duplicate key value violates unique constraint \\"inv_caboose_pkey\\"") transaction = inventory_insert_collection
42772025-04-30T06:42:51.634ZWARNtest_snapshot_basic (BackgroundTasks): inventory collection failed background_task = inventory_collection error = failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint "inv_caboose_pkey"
42782025-04-30T06:42:51.635ZDEBGtest_snapshot_basic (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 605.804727ms iteration = 8 status = {"error":"failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint \\"inv_caboose_pkey\\""}
42792025-04-30T06:42:51.635ZDEBGtest_snapshot_basic (BackgroundTasks): activating background_task = inventory_collection iteration = 9 reason = Signaled
42802025-04-30T06:42:51.635ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
42812025-04-30T06:42:51.635ZTRACtest_snapshot_basic (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
42822025-04-30T06:42:51.635ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
42832025-04-30T06:42:51.635ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
42842025-04-30T06:42:51.636ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
42852025-04-30T06:42:51.640ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
42862025-04-30T06:42:51.645ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
42872025-04-30T06:42:51.645ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
42882025-04-30T06:42:51.645ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
42892025-04-30T06:42:51.646ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
42902025-04-30T06:42:51.652ZDEBGtest_snapshot_basic (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = [(d1ad44e5-18dd-4193-a70e-bf9042f0e0e1, 0)]
42912025-04-30T06:42:51.652ZTRACtest_snapshot_basic (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
42922025-04-30T06:42:51.653ZTRACtest_snapshot_basic (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: b4 b6 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 ............._mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:54039 req_id = 4e3e250b-2775-405d-ac54-5feb5bdd60aa
42932025-04-30T06:42:51.653ZDEBGtest_snapshot_basic (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 46262,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:54039 req_id = 4e3e250b-2775-405d-ac54-5feb5bdd60aa
42942025-04-30T06:42:51.654ZDEBGtest_snapshot_basic (store): query key key = _mgs._tcp
42952025-04-30T06:42:51.654ZDEBGtest_snapshot_basic (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
42962025-04-30T06:42:51.654ZDEBGtest_snapshot_basic (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:54039 query = LowerQuery { name: LowerName(Name("_mgs._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 41502, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = 4e3e250b-2775-405d-ac54-5feb5bdd60aa
42972025-04-30T06:42:51.654ZDEBGtest_snapshot_basic (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 41502, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 4236, tv_nsec: 453525630 } })
42982025-04-30T06:42:51.654ZTRACtest_snapshot_basic (DnsResolver): lookup_service_targets: looking up SRV target
42992025-04-30T06:42:51.655ZTRACtest_snapshot_basic (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: d6 00 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e ............-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:51495 req_id = a7869361-9d96-4e2b-99cf-603379732792
43002025-04-30T06:42:51.655ZDEBGtest_snapshot_basic (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 54784,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:51495 req_id = a7869361-9d96-4e2b-99cf-603379732792
43012025-04-30T06:42:51.655ZDEBGtest_snapshot_basic (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
43022025-04-30T06:42:51.655ZDEBGtest_snapshot_basic (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:51495 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = a7869361-9d96-4e2b-99cf-603379732792
43032025-04-30T06:42:51.732ZTRACtest_snapshot_basic (DnsResolver): lookup_all_socket_v6 srv dns_name = _clickhouse-admin-keeper._tcp.control-plane.oxide.internal
43042025-04-30T06:42:51.733ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
43052025-04-30T06:42:51.733ZTRACtest_snapshot_basic (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
43062025-04-30T06:42:51.733ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
43072025-04-30T06:42:51.733ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {} }
43082025-04-30T06:42:51.734ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
43092025-04-30T06:42:51.734ZINFOtest_snapshot_basic (oximeter-agent): refreshing list of producers from Nexus collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 file = oximeter/collector/src/agent.rs:562
43102025-04-30T06:42:51.735ZDEBGtest_snapshot_basic (oximeter): client request body = None method = GET uri = http://127.0.0.1:61468/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
43112025-04-30T06:42:51.735ZTRACtest_snapshot_basic (dropshot_internal): incoming request local_addr = 127.0.0.1:61468 method = GET remote_addr = 127.0.0.1:64332 req_id = bf8ce381-4e67-474e-9f4a-8730b59130c2 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
43122025-04-30T06:42:51.735ZTRACtest_snapshot_basic (dns): buffer buffer = Length: 87 (0x57) bytes\n0000: 47 81 01 00 00 01 00 00 00 00 00 01 18 5f 63 6c G............_cl\n0010: 69 63 6b 68 6f 75 73 65 2d 61 64 6d 69 6e 2d 6b ickhouse-admin-k\n0020: 65 65 70 65 72 04 5f 74 63 70 0d 63 6f 6e 74 72 eeper._tcp.contr\n0030: 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 69 ol-plane.oxide.i\n0040: 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 04 nternal..!....).\n0050: d0 00 00 00 00 00 00 ....... kind = dns peer_addr = [::1]:61821 req_id = 47cb4fef-2912-4a94-86fa-18b6eb86ef42
43132025-04-30T06:42:51.736ZDEBGtest_snapshot_basic (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 18305,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 24,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 97,\n 100,\n 109,\n 105,\n 110,\n 45,\n 107,\n 101,\n 101,\n 112,\n 101,\n 114,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:61821 req_id = 47cb4fef-2912-4a94-86fa-18b6eb86ef42
43142025-04-30T06:42:51.736ZDEBGtest_snapshot_basic (store): query key key = _clickhouse-admin-keeper._tcp
43152025-04-30T06:42:51.736ZERROtest_snapshot_basic (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 18305, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } NXDOMAIN: no records found for name: "_clickhouse-admin-keeper._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:61821 req_id = 47cb4fef-2912-4a94-86fa-18b6eb86ef42
43162025-04-30T06:42:51.736ZTRACtest_snapshot_basic (dropshot_internal): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:61468 method = GET remote_addr = 127.0.0.1:64332 req_id = bf8ce381-4e67-474e-9f4a-8730b59130c2 resource = Database uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
43172025-04-30T06:42:51.736ZDEBGtest_snapshot_basic (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:61468 method = GET remote_addr = 127.0.0.1:64332 req_id = bf8ce381-4e67-474e-9f4a-8730b59130c2 roles = RoleSet { roles: {} } uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
43182025-04-30T06:42:51.737ZDEBGtest_snapshot_basic (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:61468 method = GET remote_addr = 127.0.0.1:64332 req_id = bf8ce381-4e67-474e-9f4a-8730b59130c2 resource = Database result = Ok(()) uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
43192025-04-30T06:42:51.739ZDEBGtest_snapshot_basic (BackgroundTasks): begin collection background_task = inventory_collection
43202025-04-30T06:42:51.739ZDEBGtest_snapshot_basic (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:41502
43212025-04-30T06:42:51.739ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/ignition
43222025-04-30T06:42:51.739ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
43232025-04-30T06:42:51.745ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
43242025-04-30T06:42:51.745ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
43252025-04-30T06:42:51.745ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {} }
43262025-04-30T06:42:51.746ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
43272025-04-30T06:42:51.747ZINFOtest_snapshot_basic (dropshot_internal): request completed latency_us = 11954 local_addr = 127.0.0.1:61468 method = GET remote_addr = 127.0.0.1:64332 req_id = bf8ce381-4e67-474e-9f4a-8730b59130c2 response_code = 200 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
43282025-04-30T06:42:51.749ZDEBGtest_snapshot_basic (oximeter): client response result = Ok(Response { url: "http://127.0.0.1:61468/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending", status: 200, headers: {"content-type": "application/json", "x-request-id": "bf8ce381-4e67-474e-9f4a-8730b59130c2", "content-length": "405", "date": "Wed, 30 Apr 2025 06:42:51 GMT"} })
43292025-04-30T06:42:51.750ZDEBGtest_snapshot_basic (oximeter): client request body = None method = GET uri = http://127.0.0.1:61468/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
43302025-04-30T06:42:51.750ZTRACtest_snapshot_basic (dropshot_internal): incoming request local_addr = 127.0.0.1:61468 method = GET remote_addr = 127.0.0.1:64332 req_id = 384b24e9-ed6e-4d6f-b30b-d384420ad2f2 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
43312025-04-30T06:42:51.751ZTRACtest_snapshot_basic (dropshot_internal): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:61468 method = GET remote_addr = 127.0.0.1:64332 req_id = 384b24e9-ed6e-4d6f-b30b-d384420ad2f2 resource = Database uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
43322025-04-30T06:42:51.751ZDEBGtest_snapshot_basic (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:61468 method = GET remote_addr = 127.0.0.1:64332 req_id = 384b24e9-ed6e-4d6f-b30b-d384420ad2f2 roles = RoleSet { roles: {} } uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
43332025-04-30T06:42:51.751ZDEBGtest_snapshot_basic (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:61468 method = GET remote_addr = 127.0.0.1:64332 req_id = 384b24e9-ed6e-4d6f-b30b-d384420ad2f2 resource = Database result = Ok(()) uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
43342025-04-30T06:42:51.751ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "ab1d52c6-d045-40b9-a10b-455cbc89aaa3", "content-length": "835", "date": "Wed, 30 Apr 2025 06:42:51 GMT"} })
43352025-04-30T06:42:51.752ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/0
43362025-04-30T06:42:51.754ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
43372025-04-30T06:42:51.754ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {} }
43382025-04-30T06:42:51.755ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
43392025-04-30T06:42:51.760ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "3cbf1b13-9e82-4668-8ccc-c0b62c578435", "content-length": "734", "date": "Wed, 30 Apr 2025 06:42:51 GMT"} })
43402025-04-30T06:42:51.761ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/0/component/sp/caboose?firmware_slot=0
43412025-04-30T06:42:51.762ZINFOtest_snapshot_basic (dropshot_internal): request completed latency_us = 11189 local_addr = 127.0.0.1:61468 method = GET remote_addr = 127.0.0.1:64332 req_id = 384b24e9-ed6e-4d6f-b30b-d384420ad2f2 response_code = 200 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
43422025-04-30T06:42:51.762ZDEBGtest_snapshot_basic (oximeter): client response result = Ok(Response { url: "http://127.0.0.1:61468/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D", status: 200, headers: {"content-type": "application/json", "x-request-id": "384b24e9-ed6e-4d6f-b30b-d384420ad2f2", "content-length": "29", "date": "Wed, 30 Apr 2025 06:42:51 GMT"} })
43432025-04-30T06:42:51.763ZTRACtest_snapshot_basic (oximeter-agent): ignoring request to update existing metric producer, since the endpoint information is the same as the existing collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = 354a52d4-3078-4706-81ce-12c32f6bbeaa
43442025-04-30T06:42:51.763ZTRACtest_snapshot_basic (oximeter-agent): ignoring request to update existing metric producer, since the endpoint information is the same as the existing collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
43452025-04-30T06:42:51.763ZINFOtest_snapshot_basic (oximeter-agent): refreshed list of producers from Nexus collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 file = oximeter/collector/src/agent.rs:637 n_current_tasks = 2 n_pruned_tasks = 0
43462025-04-30T06:42:51.766ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "513c27bb-9fac-4c42-ad65-1326e1a68f42", "content-length": "111", "date": "Wed, 30 Apr 2025 06:42:51 GMT"} })
43472025-04-30T06:42:51.766ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/0/component/sp/caboose?firmware_slot=1
43482025-04-30T06:42:51.768ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "e7d3a238-6733-441f-9804-2cfd33e57da5", "content-length": "111", "date": "Wed, 30 Apr 2025 06:42:51 GMT"} })
43492025-04-30T06:42:51.768ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/0/component/rot/caboose?firmware_slot=0
43502025-04-30T06:42:51.771ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "70e74330-f8e4-48e6-99ad-58ba04477587", "content-length": "170", "date": "Wed, 30 Apr 2025 06:42:51 GMT"} })
43512025-04-30T06:42:51.771ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/0/component/rot/caboose?firmware_slot=1
43522025-04-30T06:42:51.774ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "725dd99a-a257-431a-a4f1-4f73a65a911a", "content-length": "170", "date": "Wed, 30 Apr 2025 06:42:51 GMT"} })
43532025-04-30T06:42:51.774ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/0/component/stage0/caboose?firmware_slot=0
43542025-04-30T06:42:51.776ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "68de0962-e144-456d-a3a8-52707847d2ee", "content-length": "179", "date": "Wed, 30 Apr 2025 06:42:51 GMT"} })
43552025-04-30T06:42:51.776ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/0/component/stage0/caboose?firmware_slot=1
43562025-04-30T06:42:51.781ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "b0920090-9bc7-4325-969c-04de0eac8cc4", "content-length": "179", "date": "Wed, 30 Apr 2025 06:42:51 GMT"} })
43572025-04-30T06:42:51.781ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/0/component/rot/cmpa
43582025-04-30T06:42:51.783ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "b55d2315-d2ed-4de6-ba37-b3f2ab83da59", "content-length": "702", "date": "Wed, 30 Apr 2025 06:42:51 GMT"} })
43592025-04-30T06:42:51.783ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/0/component/rot/cfpa
43602025-04-30T06:42:51.790ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "ca65a818-0f0a-47aa-aedb-e826e173e98b", "content-length": "727", "date": "Wed, 30 Apr 2025 06:42:51 GMT"} })
43612025-04-30T06:42:51.790ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/0/component/rot/cfpa
43622025-04-30T06:42:51.791ZDEBGtest_snapshot_basic: attempt to fetch latest inventory collection result = Ok("found")
43632025-04-30T06:42:51.794ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "86d90f6d-1589-48c1-8a2b-edab8a7183cf", "content-length": "729", "date": "Wed, 30 Apr 2025 06:42:51 GMT"} })
43642025-04-30T06:42:51.794ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/0/component/rot/cfpa
43652025-04-30T06:42:51.796ZDEBGtest_snapshot_basic (collection-task): sent timer-based collection request to the collection task collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
43662025-04-30T06:42:51.797ZDEBGtest_snapshot_basic (collection-task): collecting from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
43672025-04-30T06:42:51.799ZTRACtest_snapshot_basic (collection-task): sent collection request to producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
43682025-04-30T06:42:51.800ZDEBGtest_snapshot_basic (collection-task): collected results from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 n_results = 1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
43692025-04-30T06:42:51.800ZTRACtest_snapshot_basic (collection-task): forwarded results to main collection loop collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
43702025-04-30T06:42:51.800ZTRACtest_snapshot_basic (collection-task): top of inner collection loop, waiting for next request collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
43712025-04-30T06:42:51.801ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "e4973189-dbbb-4763-8607-8ea1372328a0", "content-length": "728", "date": "Wed, 30 Apr 2025 06:42:51 GMT"} })
43722025-04-30T06:42:51.801ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/1
43732025-04-30T06:42:51.803ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "6cba05a1-8609-4339-888e-2ec7514b99a4", "content-length": "734", "date": "Wed, 30 Apr 2025 06:42:51 GMT"} })
43742025-04-30T06:42:51.803ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/1/component/sp/caboose?firmware_slot=0
43752025-04-30T06:42:51.814ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "93d0c86b-f8a0-4735-bf40-2a987885300e", "content-length": "111", "date": "Wed, 30 Apr 2025 06:42:51 GMT"} })
43762025-04-30T06:42:51.814ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/1/component/sp/caboose?firmware_slot=1
43772025-04-30T06:42:51.819ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "763d6f91-1d31-4f98-b092-b64ff0624aa6", "content-length": "111", "date": "Wed, 30 Apr 2025 06:42:51 GMT"} })
43782025-04-30T06:42:51.819ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/1/component/rot/caboose?firmware_slot=0
43792025-04-30T06:42:51.822ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "999dbe24-3fc8-40d2-b85e-326ce3ececec", "content-length": "170", "date": "Wed, 30 Apr 2025 06:42:51 GMT"} })
43802025-04-30T06:42:51.822ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/1/component/rot/caboose?firmware_slot=1
43812025-04-30T06:42:51.826ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "6a71d137-f01d-4580-901b-ca46a256c1fb", "content-length": "170", "date": "Wed, 30 Apr 2025 06:42:51 GMT"} })
43822025-04-30T06:42:51.826ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/1/component/stage0/caboose?firmware_slot=0
43832025-04-30T06:42:51.830ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "be79b449-1418-415b-a2b2-5887413acf76", "content-length": "179", "date": "Wed, 30 Apr 2025 06:42:51 GMT"} })
43842025-04-30T06:42:51.831ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/1/component/stage0/caboose?firmware_slot=1
43852025-04-30T06:42:51.834ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "5ed4a67a-a1af-42f7-a988-7de7b1340af1", "content-length": "179", "date": "Wed, 30 Apr 2025 06:42:51 GMT"} })
43862025-04-30T06:42:51.835ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/1/component/rot/cmpa
43872025-04-30T06:42:51.836ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "de107628-acae-4ef4-85cf-e70b78ccf7e7", "content-length": "702", "date": "Wed, 30 Apr 2025 06:42:51 GMT"} })
43882025-04-30T06:42:51.837ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/1/component/rot/cfpa
43892025-04-30T06:42:51.838ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "ba833d75-0265-4a8f-b3e9-6babd9989d6f", "content-length": "727", "date": "Wed, 30 Apr 2025 06:42:51 GMT"} })
43902025-04-30T06:42:51.838ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/1/component/rot/cfpa
43912025-04-30T06:42:51.854ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
43922025-04-30T06:42:51.854ZTRACtest_snapshot_basic (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
43932025-04-30T06:42:51.854ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
43942025-04-30T06:42:51.854ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {} }
43952025-04-30T06:42:51.855ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
43962025-04-30T06:42:51.857ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "707dc4ba-14ed-4892-9e8e-dd4be6b4938d", "content-length": "729", "date": "Wed, 30 Apr 2025 06:42:51 GMT"} })
43972025-04-30T06:42:51.858ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/1/component/rot/cfpa
43982025-04-30T06:42:51.858ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
43992025-04-30T06:42:51.864ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
44002025-04-30T06:42:51.864ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
44012025-04-30T06:42:51.864ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {} }
44022025-04-30T06:42:51.865ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
44032025-04-30T06:42:51.867ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "e5e98b6a-d75a-4a21-97dd-e7d0038a4b96", "content-length": "728", "date": "Wed, 30 Apr 2025 06:42:51 GMT"} })
44042025-04-30T06:42:51.867ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/0
44052025-04-30T06:42:51.868ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "a1a5ddf3-d273-40bd-9034-18d26e696b4a", "content-length": "723", "date": "Wed, 30 Apr 2025 06:42:51 GMT"} })
44062025-04-30T06:42:51.869ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/0/component/sp/caboose?firmware_slot=0
44072025-04-30T06:42:51.869ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
44082025-04-30T06:42:51.869ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {} }
44092025-04-30T06:42:51.869ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
44102025-04-30T06:42:51.872ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "11635496-8890-483b-83e5-37fde2aa88dc", "content-length": "109", "date": "Wed, 30 Apr 2025 06:42:51 GMT"} })
44112025-04-30T06:42:51.872ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/0/component/sp/caboose?firmware_slot=1
44122025-04-30T06:42:51.876ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "1f962ccc-7aaa-41ed-91f5-5616646c5d2a", "content-length": "109", "date": "Wed, 30 Apr 2025 06:42:51 GMT"} })
44132025-04-30T06:42:51.876ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/0/component/rot/caboose?firmware_slot=0
44142025-04-30T06:42:51.880ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "ad10cd45-951f-4b70-bfe8-8a1d3f8a9aaf", "content-length": "169", "date": "Wed, 30 Apr 2025 06:42:51 GMT"} })
44152025-04-30T06:42:51.880ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/0/component/rot/caboose?firmware_slot=1
44162025-04-30T06:42:51.885ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "de67d8a2-814a-4b5b-9ba3-90528629638b", "content-length": "169", "date": "Wed, 30 Apr 2025 06:42:51 GMT"} })
44172025-04-30T06:42:51.885ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/0/component/stage0/caboose?firmware_slot=0
44182025-04-30T06:42:51.887ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "bb55e6ac-db0b-4882-a3be-89a25041dcc0", "content-length": "178", "date": "Wed, 30 Apr 2025 06:42:51 GMT"} })
44192025-04-30T06:42:51.887ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/0/component/stage0/caboose?firmware_slot=1
44202025-04-30T06:42:51.891ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "1dffed9d-5a0d-4cfd-9cfc-7f52d82f2e17", "content-length": "178", "date": "Wed, 30 Apr 2025 06:42:51 GMT"} })
44212025-04-30T06:42:51.892ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/0/component/rot/cmpa
44222025-04-30T06:42:51.893ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "d5893807-b90a-4443-bc62-ffa0b0602697", "content-length": "702", "date": "Wed, 30 Apr 2025 06:42:51 GMT"} })
44232025-04-30T06:42:51.893ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/0/component/rot/cfpa
44242025-04-30T06:42:51.895ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "a60bd663-a954-47a2-be4c-3621a83bad25", "content-length": "727", "date": "Wed, 30 Apr 2025 06:42:51 GMT"} })
44252025-04-30T06:42:51.895ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/0/component/rot/cfpa
44262025-04-30T06:42:51.896ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "e9be27e9-70b5-47bb-8833-3f5d36f5cf74", "content-length": "729", "date": "Wed, 30 Apr 2025 06:42:51 GMT"} })
44272025-04-30T06:42:51.896ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/0/component/rot/cfpa
44282025-04-30T06:42:51.899ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "f9f8a81c-0fac-4895-ac89-7c5a1f7cb93a", "content-length": "728", "date": "Wed, 30 Apr 2025 06:42:51 GMT"} })
44292025-04-30T06:42:51.899ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/1
44302025-04-30T06:42:51.900ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "ebdd7b68-2711-40d7-86ff-2e5588ffb3b5", "content-length": "723", "date": "Wed, 30 Apr 2025 06:42:51 GMT"} })
44312025-04-30T06:42:51.901ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/1/component/sp/caboose?firmware_slot=0
44322025-04-30T06:42:51.905ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "06520424-3c87-4d88-bf54-e2017f9af0ee", "content-length": "109", "date": "Wed, 30 Apr 2025 06:42:51 GMT"} })
44332025-04-30T06:42:51.905ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/1/component/sp/caboose?firmware_slot=1
44342025-04-30T06:42:51.906ZDEBGtest_snapshot_basic: attempt to fetch latest inventory collection result = Ok("found")
44352025-04-30T06:42:51.908ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "4286c94e-4c93-4248-ab36-3a30d3b45e9d", "content-length": "109", "date": "Wed, 30 Apr 2025 06:42:51 GMT"} })
44362025-04-30T06:42:51.908ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/1/component/rot/caboose?firmware_slot=0
44372025-04-30T06:42:51.910ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "0aa10a5f-2aaf-4a90-b385-0b5018d34010", "content-length": "169", "date": "Wed, 30 Apr 2025 06:42:51 GMT"} })
44382025-04-30T06:42:51.910ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/1/component/rot/caboose?firmware_slot=1
44392025-04-30T06:42:51.912ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "c9bb40a4-8985-43fd-860d-c3a90139ace5", "content-length": "169", "date": "Wed, 30 Apr 2025 06:42:51 GMT"} })
44402025-04-30T06:42:51.913ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/1/component/stage0/caboose?firmware_slot=0
44412025-04-30T06:42:51.915ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "203248ac-79dc-4d3e-b647-3ee976a71f2a", "content-length": "178", "date": "Wed, 30 Apr 2025 06:42:51 GMT"} })
44422025-04-30T06:42:51.916ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/1/component/stage0/caboose?firmware_slot=1
44432025-04-30T06:42:51.919ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "58a127c8-0413-4a86-a11e-9423bd05b5eb", "content-length": "178", "date": "Wed, 30 Apr 2025 06:42:51 GMT"} })
44442025-04-30T06:42:51.919ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/1/component/rot/cmpa
44452025-04-30T06:42:51.921ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "516165fa-fe2a-4a2e-acc0-286ca9841e67", "content-length": "702", "date": "Wed, 30 Apr 2025 06:42:51 GMT"} })
44462025-04-30T06:42:51.922ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/1/component/rot/cfpa
44472025-04-30T06:42:51.924ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "b8361b85-4ae5-4dd4-981b-05e488afe2df", "content-length": "727", "date": "Wed, 30 Apr 2025 06:42:51 GMT"} })
44482025-04-30T06:42:51.924ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/1/component/rot/cfpa
44492025-04-30T06:42:51.925ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "273100be-7a42-4e08-ad46-51f468c18657", "content-length": "729", "date": "Wed, 30 Apr 2025 06:42:51 GMT"} })
44502025-04-30T06:42:51.925ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/1/component/rot/cfpa
44512025-04-30T06:42:51.928ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "65703a1b-8eed-49e5-8eaa-995e431ee2bf", "content-length": "728", "date": "Wed, 30 Apr 2025 06:42:51 GMT"} })
44522025-04-30T06:42:51.928ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
44532025-04-30T06:42:51.929ZTRACtest_snapshot_basic (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
44542025-04-30T06:42:51.929ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
44552025-04-30T06:42:51.929ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
44562025-04-30T06:42:51.929ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
44572025-04-30T06:42:51.936ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
44582025-04-30T06:42:51.940ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
44592025-04-30T06:42:51.940ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
44602025-04-30T06:42:51.940ZTRACtest_snapshot_basic (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
44612025-04-30T06:42:51.940ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
44622025-04-30T06:42:51.940ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
44632025-04-30T06:42:51.941ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
44642025-04-30T06:42:51.945ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
44652025-04-30T06:42:51.952ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
44662025-04-30T06:42:51.953ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
44672025-04-30T06:42:51.953ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
44682025-04-30T06:42:51.960ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
44692025-04-30T06:42:51.961ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
44702025-04-30T06:42:51.961ZTRACtest_snapshot_basic (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
44712025-04-30T06:42:51.961ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
44722025-04-30T06:42:51.961ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {} }
44732025-04-30T06:42:51.961ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
44742025-04-30T06:42:52.048ZDEBGtest_snapshot_basic (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:41923
44752025-04-30T06:42:52.048ZDEBGtest_snapshot_basic (BackgroundTasks): client request SledAgent = http://[::1]:41923 background_task = inventory_collection body = None method = GET uri = http://[::1]:41923/inventory
44762025-04-30T06:42:52.049ZINFOtest_snapshot_basic (dropshot): accepted connection local_addr = [::1]:41923 remote_addr = [::1]:56403 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
44772025-04-30T06:42:52.049ZDEBGtest_snapshot_basic (results-sink-cluster): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
44782025-04-30T06:42:52.049ZDEBGtest_snapshot_basic (results-sink): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
44792025-04-30T06:42:52.049ZDEBGtest_snapshot_basic (clickhouse-client): unrolling 1 total samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 6ad06c6b-9a6e-4ed1-b3f4-5d2e9da3f8a7
44802025-04-30T06:42:52.050ZTRACtest_snapshot_basic (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 6ad06c6b-9a6e-4ed1-b3f4-5d2e9da3f8a7 n_columns = 4 n_rows = 2 sql = INSERT INTO oximeter.fields_string FORMAT Native
44812025-04-30T06:42:52.051ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
44822025-04-30T06:42:52.055ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
44832025-04-30T06:42:52.055ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
44842025-04-30T06:42:52.055ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {} }
44852025-04-30T06:42:52.056ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
44862025-04-30T06:42:52.056ZTRACtest_snapshot_basic (dropshot): incoming request local_addr = [::1]:41923 method = GET remote_addr = [::1]:56403 req_id = 98492907-195b-4a7a-8279-7246911c858a sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
44872025-04-30T06:42:52.057ZINFOtest_snapshot_basic (dropshot): request completed latency_us = 1049 local_addr = [::1]:41923 method = GET remote_addr = [::1]:56403 req_id = 98492907-195b-4a7a-8279-7246911c858a response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
44882025-04-30T06:42:52.057ZDEBGtest_snapshot_basic (BackgroundTasks): client response SledAgent = http://[::1]:41923 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:41923/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "98492907-195b-4a7a-8279-7246911c858a", "content-length": "3141", "date": "Wed, 30 Apr 2025 06:42:51 GMT"} })
44892025-04-30T06:42:52.059ZDEBGtest_snapshot_basic (BackgroundTasks): begin collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
44902025-04-30T06:42:52.059ZDEBGtest_snapshot_basic (BackgroundTasks): end collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
44912025-04-30T06:42:52.059ZDEBGtest_snapshot_basic (BackgroundTasks): finished collection background_task = inventory_collection
44922025-04-30T06:42:52.061ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
44932025-04-30T06:42:52.061ZTRACtest_snapshot_basic (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
44942025-04-30T06:42:52.061ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
44952025-04-30T06:42:52.061ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
44962025-04-30T06:42:52.061ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
44972025-04-30T06:42:52.065ZDEBGtest_snapshot_basic (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 6ad06c6b-9a6e-4ed1-b3f4-5d2e9da3f8a7 n_rows = 2 table_name = fields_string
44982025-04-30T06:42:52.065ZTRACtest_snapshot_basic (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 6ad06c6b-9a6e-4ed1-b3f4-5d2e9da3f8a7 n_columns = 4 n_rows = 1 sql = INSERT INTO oximeter.measurements_i64 FORMAT Native
44992025-04-30T06:42:52.067ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
45002025-04-30T06:42:52.067ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {} }
45012025-04-30T06:42:52.068ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
45022025-04-30T06:42:52.068ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
45032025-04-30T06:42:52.072ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
45042025-04-30T06:42:52.072ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
45052025-04-30T06:42:52.072ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
45062025-04-30T06:42:52.073ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
45072025-04-30T06:42:52.077ZDEBGtest_snapshot_basic (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 6ad06c6b-9a6e-4ed1-b3f4-5d2e9da3f8a7 n_rows = 1 table_name = measurements_i64
45082025-04-30T06:42:52.077ZTRACtest_snapshot_basic (results-sink): successfully inserted samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
45092025-04-30T06:42:52.120ZWARNtest_snapshot_basic (datastore): Non-retryable transaction failure err = DatabaseError(UniqueViolation, "duplicate key value violates unique constraint \\"inv_caboose_pkey\\"") transaction = inventory_insert_collection
45102025-04-30T06:42:52.120ZWARNtest_snapshot_basic (BackgroundTasks): inventory collection failed background_task = inventory_collection error = failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint "inv_caboose_pkey"
45112025-04-30T06:42:52.120ZDEBGtest_snapshot_basic (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 485.694233ms iteration = 9 status = {"error":"failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint \\"inv_caboose_pkey\\""}
45122025-04-30T06:42:52.120ZDEBGtest_snapshot_basic (BackgroundTasks): activating background_task = inventory_collection iteration = 10 reason = Signaled
45132025-04-30T06:42:52.120ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
45142025-04-30T06:42:52.120ZTRACtest_snapshot_basic (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
45152025-04-30T06:42:52.121ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
45162025-04-30T06:42:52.121ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
45172025-04-30T06:42:52.121ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
45182025-04-30T06:42:52.125ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
45192025-04-30T06:42:52.131ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
45202025-04-30T06:42:52.131ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
45212025-04-30T06:42:52.131ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
45222025-04-30T06:42:52.131ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
45232025-04-30T06:42:52.135ZDEBGtest_snapshot_basic: attempt to fetch latest inventory collection result = Ok("found")
45242025-04-30T06:42:52.136ZDEBGtest_snapshot_basic (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = [(d1ad44e5-18dd-4193-a70e-bf9042f0e0e1, 0)]
45252025-04-30T06:42:52.136ZTRACtest_snapshot_basic (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
45262025-04-30T06:42:52.137ZTRACtest_snapshot_basic (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: 9f ef 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 ............._mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:64012 req_id = 7231e3be-23c3-4fc8-9e6b-20e1aa5e9969
45272025-04-30T06:42:52.137ZDEBGtest_snapshot_basic (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 40943,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:64012 req_id = 7231e3be-23c3-4fc8-9e6b-20e1aa5e9969
45282025-04-30T06:42:52.137ZDEBGtest_snapshot_basic (store): query key key = _mgs._tcp
45292025-04-30T06:42:52.138ZDEBGtest_snapshot_basic (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
45302025-04-30T06:42:52.138ZDEBGtest_snapshot_basic (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:64012 query = LowerQuery { name: LowerName(Name("_mgs._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 41502, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = 7231e3be-23c3-4fc8-9e6b-20e1aa5e9969
45312025-04-30T06:42:52.139ZDEBGtest_snapshot_basic (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 41502, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 4236, tv_nsec: 937925955 } })
45322025-04-30T06:42:52.139ZTRACtest_snapshot_basic (DnsResolver): lookup_service_targets: looking up SRV target
45332025-04-30T06:42:52.139ZTRACtest_snapshot_basic (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: 54 aa 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e T...........-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:63432 req_id = 423631e8-4df1-4e41-8040-23c8b551617d
45342025-04-30T06:42:52.139ZDEBGtest_snapshot_basic (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 21674,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:63432 req_id = 423631e8-4df1-4e41-8040-23c8b551617d
45352025-04-30T06:42:52.139ZDEBGtest_snapshot_basic (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
45362025-04-30T06:42:52.139ZDEBGtest_snapshot_basic (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:63432 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = 423631e8-4df1-4e41-8040-23c8b551617d
45372025-04-30T06:42:52.194ZTRACtest_snapshot_basic (DnsResolver): lookup_all_socket_v6 srv dns_name = _clickhouse-admin-keeper._tcp.control-plane.oxide.internal
45382025-04-30T06:42:52.194ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
45392025-04-30T06:42:52.194ZTRACtest_snapshot_basic (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
45402025-04-30T06:42:52.194ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
45412025-04-30T06:42:52.194ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {} }
45422025-04-30T06:42:52.195ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
45432025-04-30T06:42:52.195ZWARNtest_snapshot_basic (results-sink-cluster): failed to insert some results into metric DB collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 error = Failed to check out connection to database: No backends found for this service file = oximeter/collector/src/results_sink.rs:92
45442025-04-30T06:42:52.196ZTRACtest_snapshot_basic (dns): buffer buffer = Length: 87 (0x57) bytes\n0000: 17 db 01 00 00 01 00 00 00 00 00 01 18 5f 63 6c ............._cl\n0010: 69 63 6b 68 6f 75 73 65 2d 61 64 6d 69 6e 2d 6b ickhouse-admin-k\n0020: 65 65 70 65 72 04 5f 74 63 70 0d 63 6f 6e 74 72 eeper._tcp.contr\n0030: 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 69 ol-plane.oxide.i\n0040: 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 04 nternal..!....).\n0050: d0 00 00 00 00 00 00 ....... kind = dns peer_addr = [::1]:52770 req_id = fcab4799-6114-4aba-9fc7-db334ce6ac26
45452025-04-30T06:42:52.196ZDEBGtest_snapshot_basic (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 6107,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 24,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 97,\n 100,\n 109,\n 105,\n 110,\n 45,\n 107,\n 101,\n 101,\n 112,\n 101,\n 114,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:52770 req_id = fcab4799-6114-4aba-9fc7-db334ce6ac26
45462025-04-30T06:42:52.196ZDEBGtest_snapshot_basic (store): query key key = _clickhouse-admin-keeper._tcp
45472025-04-30T06:42:52.196ZERROtest_snapshot_basic (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 6107, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } NXDOMAIN: no records found for name: "_clickhouse-admin-keeper._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:52770 req_id = fcab4799-6114-4aba-9fc7-db334ce6ac26
45482025-04-30T06:42:52.196ZDEBGtest_snapshot_basic (BackgroundTasks): begin collection background_task = inventory_collection
45492025-04-30T06:42:52.196ZDEBGtest_snapshot_basic (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:41502
45502025-04-30T06:42:52.196ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/ignition
45512025-04-30T06:42:52.199ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
45522025-04-30T06:42:52.205ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
45532025-04-30T06:42:52.205ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
45542025-04-30T06:42:52.206ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {} }
45552025-04-30T06:42:52.207ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
45562025-04-30T06:42:52.208ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "fc2723f5-27d2-4aa3-8688-00d1b92bcbf0", "content-length": "835", "date": "Wed, 30 Apr 2025 06:42:52 GMT"} })
45572025-04-30T06:42:52.209ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/0
45582025-04-30T06:42:52.212ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
45592025-04-30T06:42:52.212ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {} }
45602025-04-30T06:42:52.213ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
45612025-04-30T06:42:52.217ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "ccb4158f-5900-4b6c-8dac-a3e8c3315ffd", "content-length": "734", "date": "Wed, 30 Apr 2025 06:42:52 GMT"} })
45622025-04-30T06:42:52.217ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/0/component/sp/caboose?firmware_slot=0
45632025-04-30T06:42:52.222ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "24145f21-7aea-43a4-a753-7c170cc00516", "content-length": "111", "date": "Wed, 30 Apr 2025 06:42:52 GMT"} })
45642025-04-30T06:42:52.223ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/0/component/sp/caboose?firmware_slot=1
45652025-04-30T06:42:52.237ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "98ea16f9-ebcf-47ef-a1ae-ad7959887351", "content-length": "111", "date": "Wed, 30 Apr 2025 06:42:52 GMT"} })
45662025-04-30T06:42:52.237ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/0/component/rot/caboose?firmware_slot=0
45672025-04-30T06:42:52.241ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "20b6cb87-92c4-4e29-b699-e9d8d344bc79", "content-length": "170", "date": "Wed, 30 Apr 2025 06:42:52 GMT"} })
45682025-04-30T06:42:52.241ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/0/component/rot/caboose?firmware_slot=1
45692025-04-30T06:42:52.252ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "5d638294-8daf-4bea-b9f5-5703e8b6f0d1", "content-length": "170", "date": "Wed, 30 Apr 2025 06:42:52 GMT"} })
45702025-04-30T06:42:52.253ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/0/component/stage0/caboose?firmware_slot=0
45712025-04-30T06:42:52.257ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "92007e9a-a540-415d-a983-0e6242d20049", "content-length": "179", "date": "Wed, 30 Apr 2025 06:42:52 GMT"} })
45722025-04-30T06:42:52.257ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/0/component/stage0/caboose?firmware_slot=1
45732025-04-30T06:42:52.259ZDEBGtest_snapshot_basic: attempt to fetch latest inventory collection result = Ok("found")
45742025-04-30T06:42:52.262ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "22efae21-31e6-4384-b57c-5163ca985f7f", "content-length": "179", "date": "Wed, 30 Apr 2025 06:42:52 GMT"} })
45752025-04-30T06:42:52.262ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/0/component/rot/cmpa
45762025-04-30T06:42:52.263ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "1e12c2da-aa69-453a-ac2c-874c3d4ae347", "content-length": "702", "date": "Wed, 30 Apr 2025 06:42:52 GMT"} })
45772025-04-30T06:42:52.263ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/0/component/rot/cfpa
45782025-04-30T06:42:52.265ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "9a4ac762-2f96-4f57-a4a5-d11bd0348c9f", "content-length": "727", "date": "Wed, 30 Apr 2025 06:42:52 GMT"} })
45792025-04-30T06:42:52.265ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/0/component/rot/cfpa
45802025-04-30T06:42:52.266ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "f9258215-5ed5-4922-90f5-1e9a80b085ed", "content-length": "729", "date": "Wed, 30 Apr 2025 06:42:52 GMT"} })
45812025-04-30T06:42:52.266ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/0/component/rot/cfpa
45822025-04-30T06:42:52.267ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "58717baa-14bf-4f96-a158-a4ab52916196", "content-length": "728", "date": "Wed, 30 Apr 2025 06:42:52 GMT"} })
45832025-04-30T06:42:52.267ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/1
45842025-04-30T06:42:52.268ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "bea7f1b8-4140-401b-86f8-d0bee7aa4467", "content-length": "734", "date": "Wed, 30 Apr 2025 06:42:52 GMT"} })
45852025-04-30T06:42:52.268ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/1/component/sp/caboose?firmware_slot=0
45862025-04-30T06:42:52.270ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "12ab8d72-581c-42f2-a452-529b94cea48a", "content-length": "111", "date": "Wed, 30 Apr 2025 06:42:52 GMT"} })
45872025-04-30T06:42:52.270ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/1/component/sp/caboose?firmware_slot=1
45882025-04-30T06:42:52.272ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "f372863d-0284-4ed5-ac39-227a57f36249", "content-length": "111", "date": "Wed, 30 Apr 2025 06:42:52 GMT"} })
45892025-04-30T06:42:52.272ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/1/component/rot/caboose?firmware_slot=0
45902025-04-30T06:42:52.275ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "62e95f70-ee67-4dc8-b137-c4eda8937ada", "content-length": "170", "date": "Wed, 30 Apr 2025 06:42:52 GMT"} })
45912025-04-30T06:42:52.275ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/1/component/rot/caboose?firmware_slot=1
45922025-04-30T06:42:52.284ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "8c4614af-7ab5-4491-b8d9-84a15739f2bb", "content-length": "170", "date": "Wed, 30 Apr 2025 06:42:52 GMT"} })
45932025-04-30T06:42:52.284ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/1/component/stage0/caboose?firmware_slot=0
45942025-04-30T06:42:52.287ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "d6ec1227-3a40-42ec-ab6b-41acb3a44f9f", "content-length": "179", "date": "Wed, 30 Apr 2025 06:42:52 GMT"} })
45952025-04-30T06:42:52.287ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/1/component/stage0/caboose?firmware_slot=1
45962025-04-30T06:42:52.289ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "87079c16-204b-44a2-bbf2-8e71024f7d76", "content-length": "179", "date": "Wed, 30 Apr 2025 06:42:52 GMT"} })
45972025-04-30T06:42:52.289ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/1/component/rot/cmpa
45982025-04-30T06:42:52.291ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "92922c97-4523-446b-a91c-6b78ac8a0cc6", "content-length": "702", "date": "Wed, 30 Apr 2025 06:42:52 GMT"} })
45992025-04-30T06:42:52.291ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/1/component/rot/cfpa
46002025-04-30T06:42:52.292ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "c4b5db9d-7440-48ed-8e31-50d919b69582", "content-length": "727", "date": "Wed, 30 Apr 2025 06:42:52 GMT"} })
46012025-04-30T06:42:52.293ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/1/component/rot/cfpa
46022025-04-30T06:42:52.295ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "90badcc1-1140-429a-b027-6889dda0d977", "content-length": "729", "date": "Wed, 30 Apr 2025 06:42:52 GMT"} })
46032025-04-30T06:42:52.295ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/1/component/rot/cfpa
46042025-04-30T06:42:52.296ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "95c033ce-02c4-4405-8343-f5c879368f0e", "content-length": "728", "date": "Wed, 30 Apr 2025 06:42:52 GMT"} })
46052025-04-30T06:42:52.296ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/0
46062025-04-30T06:42:52.297ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "7b5d39ec-55a9-436b-9160-392d56770ad8", "content-length": "723", "date": "Wed, 30 Apr 2025 06:42:52 GMT"} })
46072025-04-30T06:42:52.297ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/0/component/sp/caboose?firmware_slot=0
46082025-04-30T06:42:52.300ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "c8f2655d-3f1e-46ce-9879-123fb95a0ecc", "content-length": "109", "date": "Wed, 30 Apr 2025 06:42:52 GMT"} })
46092025-04-30T06:42:52.300ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/0/component/sp/caboose?firmware_slot=1
46102025-04-30T06:42:52.301ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "3ead2e59-7f8c-4a59-9067-e40857837ed3", "content-length": "109", "date": "Wed, 30 Apr 2025 06:42:52 GMT"} })
46112025-04-30T06:42:52.301ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/0/component/rot/caboose?firmware_slot=0
46122025-04-30T06:42:52.303ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "e11133a3-b143-4283-96f6-61ca1e633c01", "content-length": "169", "date": "Wed, 30 Apr 2025 06:42:52 GMT"} })
46132025-04-30T06:42:52.303ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/0/component/rot/caboose?firmware_slot=1
46142025-04-30T06:42:52.304ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "e759476d-9107-45bf-ba1e-0afddbf050b1", "content-length": "169", "date": "Wed, 30 Apr 2025 06:42:52 GMT"} })
46152025-04-30T06:42:52.305ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/0/component/stage0/caboose?firmware_slot=0
46162025-04-30T06:42:52.308ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "f91d19f9-5e72-4db0-9bdc-01dc909504f8", "content-length": "178", "date": "Wed, 30 Apr 2025 06:42:52 GMT"} })
46172025-04-30T06:42:52.308ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/0/component/stage0/caboose?firmware_slot=1
46182025-04-30T06:42:52.311ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
46192025-04-30T06:42:52.311ZTRACtest_snapshot_basic (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
46202025-04-30T06:42:52.311ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
46212025-04-30T06:42:52.311ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {} }
46222025-04-30T06:42:52.312ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
46232025-04-30T06:42:52.313ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "0650af6c-d814-4a87-a210-b99af242bbd1", "content-length": "178", "date": "Wed, 30 Apr 2025 06:42:52 GMT"} })
46242025-04-30T06:42:52.314ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/0/component/rot/cmpa
46252025-04-30T06:42:52.319ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
46262025-04-30T06:42:52.323ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
46272025-04-30T06:42:52.323ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
46282025-04-30T06:42:52.323ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {} }
46292025-04-30T06:42:52.323ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
46302025-04-30T06:42:52.326ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "1e158832-ba6d-4a6b-8e06-f315de38ab24", "content-length": "702", "date": "Wed, 30 Apr 2025 06:42:52 GMT"} })
46312025-04-30T06:42:52.326ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/0/component/rot/cfpa
46322025-04-30T06:42:52.326ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
46332025-04-30T06:42:52.326ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {} }
46342025-04-30T06:42:52.327ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
46352025-04-30T06:42:52.329ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "34a68650-acde-4734-9034-dd04e6c52489", "content-length": "727", "date": "Wed, 30 Apr 2025 06:42:52 GMT"} })
46362025-04-30T06:42:52.329ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/0/component/rot/cfpa
46372025-04-30T06:42:52.330ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "dec0e7f5-f47a-493f-a2f8-34fb1a77bc28", "content-length": "729", "date": "Wed, 30 Apr 2025 06:42:52 GMT"} })
46382025-04-30T06:42:52.330ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/0/component/rot/cfpa
46392025-04-30T06:42:52.333ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "6416d63e-e418-4ec3-b8df-58b8512e56ae", "content-length": "728", "date": "Wed, 30 Apr 2025 06:42:52 GMT"} })
46402025-04-30T06:42:52.333ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/1
46412025-04-30T06:42:52.336ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "52f4fd56-7394-4956-b76b-eb2a4e677520", "content-length": "723", "date": "Wed, 30 Apr 2025 06:42:52 GMT"} })
46422025-04-30T06:42:52.336ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/1/component/sp/caboose?firmware_slot=0
46432025-04-30T06:42:52.340ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "4a613ecd-dcb0-435e-a849-65cbac34be6e", "content-length": "109", "date": "Wed, 30 Apr 2025 06:42:52 GMT"} })
46442025-04-30T06:42:52.340ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/1/component/sp/caboose?firmware_slot=1
46452025-04-30T06:42:52.344ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "3cd0ea83-9bf8-4776-92b0-0f11b4038ae6", "content-length": "109", "date": "Wed, 30 Apr 2025 06:42:52 GMT"} })
46462025-04-30T06:42:52.344ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/1/component/rot/caboose?firmware_slot=0
46472025-04-30T06:42:52.378ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "02c6ce2f-01e5-43c8-9b37-8d8cc02cb24d", "content-length": "169", "date": "Wed, 30 Apr 2025 06:42:52 GMT"} })
46482025-04-30T06:42:52.379ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/1/component/rot/caboose?firmware_slot=1
46492025-04-30T06:42:52.381ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "4dd4ed7f-c972-4d8b-8190-792cc7530bd8", "content-length": "169", "date": "Wed, 30 Apr 2025 06:42:52 GMT"} })
46502025-04-30T06:42:52.381ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/1/component/stage0/caboose?firmware_slot=0
46512025-04-30T06:42:52.383ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "898f813b-7887-4342-9102-b3e2a88c14d8", "content-length": "178", "date": "Wed, 30 Apr 2025 06:42:52 GMT"} })
46522025-04-30T06:42:52.383ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/1/component/stage0/caboose?firmware_slot=1
46532025-04-30T06:42:52.386ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "d27df8f0-2024-4744-9b29-73c77fd6defa", "content-length": "178", "date": "Wed, 30 Apr 2025 06:42:52 GMT"} })
46542025-04-30T06:42:52.386ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/1/component/rot/cmpa
46552025-04-30T06:42:52.387ZDEBGtest_snapshot_basic: attempt to fetch latest inventory collection result = Ok("found")
46562025-04-30T06:42:52.388ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "fee4a3c8-d691-4f38-ba72-0195e9718807", "content-length": "702", "date": "Wed, 30 Apr 2025 06:42:52 GMT"} })
46572025-04-30T06:42:52.388ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/1/component/rot/cfpa
46582025-04-30T06:42:52.389ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "e27b7deb-b1d5-4f50-abf1-750b331e23ad", "content-length": "727", "date": "Wed, 30 Apr 2025 06:42:52 GMT"} })
46592025-04-30T06:42:52.389ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/1/component/rot/cfpa
46602025-04-30T06:42:52.391ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "b535af30-ee5a-4959-aefa-25626c6804e4", "content-length": "729", "date": "Wed, 30 Apr 2025 06:42:52 GMT"} })
46612025-04-30T06:42:52.391ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/1/component/rot/cfpa
46622025-04-30T06:42:52.392ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "e6d47a96-85c9-4aea-bddb-7ecb7f02d5ab", "content-length": "728", "date": "Wed, 30 Apr 2025 06:42:52 GMT"} })
46632025-04-30T06:42:52.393ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
46642025-04-30T06:42:52.393ZTRACtest_snapshot_basic (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
46652025-04-30T06:42:52.393ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
46662025-04-30T06:42:52.393ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
46672025-04-30T06:42:52.394ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
46682025-04-30T06:42:52.401ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
46692025-04-30T06:42:52.408ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
46702025-04-30T06:42:52.408ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
46712025-04-30T06:42:52.408ZTRACtest_snapshot_basic (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
46722025-04-30T06:42:52.408ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
46732025-04-30T06:42:52.408ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
46742025-04-30T06:42:52.408ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
46752025-04-30T06:42:52.411ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
46762025-04-30T06:42:52.416ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
46772025-04-30T06:42:52.416ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
46782025-04-30T06:42:52.416ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
46792025-04-30T06:42:52.417ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
46802025-04-30T06:42:52.503ZDEBGtest_snapshot_basic (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:41923
46812025-04-30T06:42:52.503ZDEBGtest_snapshot_basic (BackgroundTasks): client request SledAgent = http://[::1]:41923 background_task = inventory_collection body = None method = GET uri = http://[::1]:41923/inventory
46822025-04-30T06:42:52.504ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
46832025-04-30T06:42:52.505ZTRACtest_snapshot_basic (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
46842025-04-30T06:42:52.505ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
46852025-04-30T06:42:52.505ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {} }
46862025-04-30T06:42:52.506ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
46872025-04-30T06:42:52.506ZINFOtest_snapshot_basic (dropshot): accepted connection local_addr = [::1]:41923 remote_addr = [::1]:40772 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
46882025-04-30T06:42:52.507ZTRACtest_snapshot_basic (dropshot): incoming request local_addr = [::1]:41923 method = GET remote_addr = [::1]:40772 req_id = adedf707-5ea0-4885-9510-54c43fba1a2d sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
46892025-04-30T06:42:52.508ZINFOtest_snapshot_basic (dropshot): request completed latency_us = 873 local_addr = [::1]:41923 method = GET remote_addr = [::1]:40772 req_id = adedf707-5ea0-4885-9510-54c43fba1a2d response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
46902025-04-30T06:42:52.508ZDEBGtest_snapshot_basic (BackgroundTasks): client response SledAgent = http://[::1]:41923 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:41923/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "adedf707-5ea0-4885-9510-54c43fba1a2d", "content-length": "3141", "date": "Wed, 30 Apr 2025 06:42:52 GMT"} })
46912025-04-30T06:42:52.512ZDEBGtest_snapshot_basic (BackgroundTasks): begin collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
46922025-04-30T06:42:52.512ZDEBGtest_snapshot_basic (BackgroundTasks): end collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
46932025-04-30T06:42:52.512ZDEBGtest_snapshot_basic (BackgroundTasks): finished collection background_task = inventory_collection
46942025-04-30T06:42:52.515ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
46952025-04-30T06:42:52.515ZTRACtest_snapshot_basic (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
46962025-04-30T06:42:52.515ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
46972025-04-30T06:42:52.515ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
46982025-04-30T06:42:52.515ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
46992025-04-30T06:42:52.516ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
47002025-04-30T06:42:52.526ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
47012025-04-30T06:42:52.527ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
47022025-04-30T06:42:52.527ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {} }
47032025-04-30T06:42:52.527ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
47042025-04-30T06:42:52.531ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
47052025-04-30T06:42:52.538ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
47062025-04-30T06:42:52.538ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
47072025-04-30T06:42:52.538ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
47082025-04-30T06:42:52.538ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
47092025-04-30T06:42:52.553ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
47102025-04-30T06:42:52.553ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {} }
47112025-04-30T06:42:52.554ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
47122025-04-30T06:42:52.704ZWARNtest_snapshot_basic (datastore): Non-retryable transaction failure err = DatabaseError(UniqueViolation, "duplicate key value violates unique constraint \\"inv_caboose_pkey\\"") transaction = inventory_insert_collection
47132025-04-30T06:42:52.705ZWARNtest_snapshot_basic (BackgroundTasks): inventory collection failed background_task = inventory_collection error = failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint "inv_caboose_pkey"
47142025-04-30T06:42:52.705ZDEBGtest_snapshot_basic (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 585.043071ms iteration = 10 status = {"error":"failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint \\"inv_caboose_pkey\\""}
47152025-04-30T06:42:52.705ZDEBGtest_snapshot_basic (BackgroundTasks): activating background_task = inventory_collection iteration = 11 reason = Signaled
47162025-04-30T06:42:52.706ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
47172025-04-30T06:42:52.706ZTRACtest_snapshot_basic (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
47182025-04-30T06:42:52.706ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
47192025-04-30T06:42:52.706ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
47202025-04-30T06:42:52.708ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
47212025-04-30T06:42:52.745ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
47222025-04-30T06:42:52.751ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
47232025-04-30T06:42:52.751ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
47242025-04-30T06:42:52.751ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
47252025-04-30T06:42:52.752ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
47262025-04-30T06:42:52.753ZDEBGtest_snapshot_basic: attempt to fetch latest inventory collection result = Ok("found")
47272025-04-30T06:42:52.756ZDEBGtest_snapshot_basic (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = [(d1ad44e5-18dd-4193-a70e-bf9042f0e0e1, 0)]
47282025-04-30T06:42:52.756ZTRACtest_snapshot_basic (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
47292025-04-30T06:42:52.757ZTRACtest_snapshot_basic (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: 71 3e 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 q>..........._mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:52491 req_id = 9c06ba26-3a60-4b9d-a08c-ee935718aa59
47302025-04-30T06:42:52.757ZDEBGtest_snapshot_basic (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 28990,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:52491 req_id = 9c06ba26-3a60-4b9d-a08c-ee935718aa59
47312025-04-30T06:42:52.757ZDEBGtest_snapshot_basic (store): query key key = _mgs._tcp
47322025-04-30T06:42:52.761ZDEBGtest_snapshot_basic (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
47332025-04-30T06:42:52.761ZDEBGtest_snapshot_basic (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:52491 query = LowerQuery { name: LowerName(Name("_mgs._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 41502, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = 9c06ba26-3a60-4b9d-a08c-ee935718aa59
47342025-04-30T06:42:52.762ZDEBGtest_snapshot_basic (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 41502, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 4237, tv_nsec: 560883928 } })
47352025-04-30T06:42:52.762ZTRACtest_snapshot_basic (DnsResolver): lookup_service_targets: looking up SRV target
47362025-04-30T06:42:52.762ZTRACtest_snapshot_basic (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: c9 49 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e .I..........-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:61375 req_id = 9cabc4a5-38e2-4bae-b668-569d3f804bf9
47372025-04-30T06:42:52.762ZDEBGtest_snapshot_basic (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 51529,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:61375 req_id = 9cabc4a5-38e2-4bae-b668-569d3f804bf9
47382025-04-30T06:42:52.762ZDEBGtest_snapshot_basic (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
47392025-04-30T06:42:52.762ZDEBGtest_snapshot_basic (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:61375 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = 9cabc4a5-38e2-4bae-b668-569d3f804bf9
47402025-04-30T06:42:52.857ZTRACtest_snapshot_basic (DnsResolver): lookup_all_socket_v6 srv dns_name = _clickhouse-admin-keeper._tcp.control-plane.oxide.internal
47412025-04-30T06:42:52.857ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
47422025-04-30T06:42:52.857ZTRACtest_snapshot_basic (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
47432025-04-30T06:42:52.858ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
47442025-04-30T06:42:52.858ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {} }
47452025-04-30T06:42:52.859ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
47462025-04-30T06:42:52.859ZDEBGtest_snapshot_basic (collection-task): sent timer-based collection request to the collection task collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
47472025-04-30T06:42:52.859ZDEBGtest_snapshot_basic (collection-task): collecting from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
47482025-04-30T06:42:52.861ZTRACtest_snapshot_basic (dns): buffer buffer = Length: 87 (0x57) bytes\n0000: 4c 3b 01 00 00 01 00 00 00 00 00 01 18 5f 63 6c L;..........._cl\n0010: 69 63 6b 68 6f 75 73 65 2d 61 64 6d 69 6e 2d 6b ickhouse-admin-k\n0020: 65 65 70 65 72 04 5f 74 63 70 0d 63 6f 6e 74 72 eeper._tcp.contr\n0030: 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 69 ol-plane.oxide.i\n0040: 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 04 nternal..!....).\n0050: d0 00 00 00 00 00 00 ....... kind = dns peer_addr = [::1]:56612 req_id = 70a26266-bda7-4cc6-ab43-070763cbd6ae
47492025-04-30T06:42:52.861ZDEBGtest_snapshot_basic (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 19515,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 24,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 97,\n 100,\n 109,\n 105,\n 110,\n 45,\n 107,\n 101,\n 101,\n 112,\n 101,\n 114,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:56612 req_id = 70a26266-bda7-4cc6-ab43-070763cbd6ae
47502025-04-30T06:42:52.862ZDEBGtest_snapshot_basic (store): query key key = _clickhouse-admin-keeper._tcp
47512025-04-30T06:42:52.862ZERROtest_snapshot_basic (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 19515, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } NXDOMAIN: no records found for name: "_clickhouse-admin-keeper._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:56612 req_id = 70a26266-bda7-4cc6-ab43-070763cbd6ae
47522025-04-30T06:42:52.864ZDEBGtest_snapshot_basic (BackgroundTasks): begin collection background_task = inventory_collection
47532025-04-30T06:42:52.864ZDEBGtest_snapshot_basic (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:41502
47542025-04-30T06:42:52.864ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/ignition
47552025-04-30T06:42:52.865ZTRACtest_snapshot_basic (collection-task): sent collection request to producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
47562025-04-30T06:42:52.866ZDEBGtest_snapshot_basic (collection-task): collected results from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 n_results = 1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
47572025-04-30T06:42:52.866ZTRACtest_snapshot_basic (collection-task): forwarded results to main collection loop collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
47582025-04-30T06:42:52.866ZTRACtest_snapshot_basic (collection-task): top of inner collection loop, waiting for next request collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
47592025-04-30T06:42:52.866ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
47602025-04-30T06:42:52.873ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
47612025-04-30T06:42:52.876ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
47622025-04-30T06:42:52.878ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {} }
47632025-04-30T06:42:52.879ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
47642025-04-30T06:42:52.887ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
47652025-04-30T06:42:52.887ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {} }
47662025-04-30T06:42:52.887ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
47672025-04-30T06:42:52.888ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "794e6548-99f1-4ea1-97ca-e8523a45b8e3", "content-length": "835", "date": "Wed, 30 Apr 2025 06:42:52 GMT"} })
47682025-04-30T06:42:52.888ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/0
47692025-04-30T06:42:52.948ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "9c0dfc86-fdbb-42b2-ac83-bc233b87af9b", "content-length": "734", "date": "Wed, 30 Apr 2025 06:42:52 GMT"} })
47702025-04-30T06:42:52.953ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/0/component/sp/caboose?firmware_slot=0
47712025-04-30T06:42:52.972ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "ebe44c82-4cbc-44fa-a77b-6ddf55a07c7f", "content-length": "111", "date": "Wed, 30 Apr 2025 06:42:52 GMT"} })
47722025-04-30T06:42:52.973ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/0/component/sp/caboose?firmware_slot=1
47732025-04-30T06:42:52.979ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "a6f1eb40-ea90-4c26-9e8e-fa3e14c2df17", "content-length": "111", "date": "Wed, 30 Apr 2025 06:42:52 GMT"} })
47742025-04-30T06:42:52.979ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/0/component/rot/caboose?firmware_slot=0
47752025-04-30T06:42:53.032ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "57a41f18-8f91-4399-a414-ca53fe3f9ac1", "content-length": "170", "date": "Wed, 30 Apr 2025 06:42:52 GMT"} })
47762025-04-30T06:42:53.036ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/0/component/rot/caboose?firmware_slot=1
47772025-04-30T06:42:53.036ZDEBGtest_snapshot_basic: attempt to fetch latest inventory collection result = Ok("found")
47782025-04-30T06:42:53.037ZDEBGtest_snapshot_basic (results-sink): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
47792025-04-30T06:42:53.037ZDEBGtest_snapshot_basic (results-sink-cluster): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
47802025-04-30T06:42:53.038ZDEBGtest_snapshot_basic (clickhouse-client): unrolling 1 total samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 6ad06c6b-9a6e-4ed1-b3f4-5d2e9da3f8a7
47812025-04-30T06:42:53.046ZTRACtest_snapshot_basic (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 6ad06c6b-9a6e-4ed1-b3f4-5d2e9da3f8a7 n_columns = 4 n_rows = 2 sql = INSERT INTO oximeter.fields_string FORMAT Native
47822025-04-30T06:42:53.087ZDEBGtest_snapshot_basic (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 6ad06c6b-9a6e-4ed1-b3f4-5d2e9da3f8a7 n_rows = 2 table_name = fields_string
47832025-04-30T06:42:53.087ZTRACtest_snapshot_basic (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 6ad06c6b-9a6e-4ed1-b3f4-5d2e9da3f8a7 n_columns = 4 n_rows = 1 sql = INSERT INTO oximeter.measurements_i64 FORMAT Native
47842025-04-30T06:42:53.089ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
47852025-04-30T06:42:53.090ZTRACtest_snapshot_basic (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
47862025-04-30T06:42:53.090ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
47872025-04-30T06:42:53.090ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {} }
47882025-04-30T06:42:53.092ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
47892025-04-30T06:42:53.096ZDEBGtest_snapshot_basic (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 6ad06c6b-9a6e-4ed1-b3f4-5d2e9da3f8a7 n_rows = 1 table_name = measurements_i64
47902025-04-30T06:42:53.096ZTRACtest_snapshot_basic (results-sink): successfully inserted samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
47912025-04-30T06:42:53.097ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "6f297712-b933-4793-bb51-d5e3d4a5c271", "content-length": "170", "date": "Wed, 30 Apr 2025 06:42:52 GMT"} })
47922025-04-30T06:42:53.097ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/0/component/stage0/caboose?firmware_slot=0
47932025-04-30T06:42:53.097ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
47942025-04-30T06:42:53.103ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
47952025-04-30T06:42:53.103ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
47962025-04-30T06:42:53.103ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {} }
47972025-04-30T06:42:53.104ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
47982025-04-30T06:42:53.107ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
47992025-04-30T06:42:53.107ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {} }
48002025-04-30T06:42:53.108ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
48012025-04-30T06:42:53.110ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "9519dac9-c0a7-4f99-bdfd-e72bc50e3c5b", "content-length": "179", "date": "Wed, 30 Apr 2025 06:42:52 GMT"} })
48022025-04-30T06:42:53.110ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/0/component/stage0/caboose?firmware_slot=1
48032025-04-30T06:42:53.113ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "5796e9b4-b2a2-4764-b22b-ee7d8dc1e744", "content-length": "179", "date": "Wed, 30 Apr 2025 06:42:52 GMT"} })
48042025-04-30T06:42:53.113ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/0/component/rot/cmpa
48052025-04-30T06:42:53.116ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "da97caf1-e954-4407-974c-1613b480bb6f", "content-length": "702", "date": "Wed, 30 Apr 2025 06:42:52 GMT"} })
48062025-04-30T06:42:53.116ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/0/component/rot/cfpa
48072025-04-30T06:42:53.118ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "181e6072-39a2-46be-8c4f-91d54d22dc66", "content-length": "727", "date": "Wed, 30 Apr 2025 06:42:52 GMT"} })
48082025-04-30T06:42:53.118ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/0/component/rot/cfpa
48092025-04-30T06:42:53.120ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "fb788150-bf40-4253-ba04-628cd5b52f96", "content-length": "729", "date": "Wed, 30 Apr 2025 06:42:52 GMT"} })
48102025-04-30T06:42:53.120ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/0/component/rot/cfpa
48112025-04-30T06:42:53.122ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "35a0f644-69b2-4539-9c62-0976f585f711", "content-length": "728", "date": "Wed, 30 Apr 2025 06:42:52 GMT"} })
48122025-04-30T06:42:53.122ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/1
48132025-04-30T06:42:53.127ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "468b48fe-a647-4afe-8003-182e39d59db1", "content-length": "734", "date": "Wed, 30 Apr 2025 06:42:52 GMT"} })
48142025-04-30T06:42:53.127ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/1/component/sp/caboose?firmware_slot=0
48152025-04-30T06:42:53.129ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "2f1c37c5-6025-4b70-893b-27c03e7e0620", "content-length": "111", "date": "Wed, 30 Apr 2025 06:42:52 GMT"} })
48162025-04-30T06:42:53.129ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/1/component/sp/caboose?firmware_slot=1
48172025-04-30T06:42:53.132ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "a53043ba-5e5a-4f6c-9af3-490b0ba702f9", "content-length": "111", "date": "Wed, 30 Apr 2025 06:42:52 GMT"} })
48182025-04-30T06:42:53.133ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/1/component/rot/caboose?firmware_slot=0
48192025-04-30T06:42:53.136ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "58de9775-2a93-4f0f-b6f2-b3d9ece59e1c", "content-length": "170", "date": "Wed, 30 Apr 2025 06:42:52 GMT"} })
48202025-04-30T06:42:53.136ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/1/component/rot/caboose?firmware_slot=1
48212025-04-30T06:42:53.138ZWARNtest_snapshot_basic (results-sink-cluster): failed to insert some results into metric DB collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 error = Failed to check out connection to database: No backends found for this service file = oximeter/collector/src/results_sink.rs:92
48222025-04-30T06:42:53.142ZDEBGtest_snapshot_basic: attempt to fetch latest inventory collection result = Ok("found")
48232025-04-30T06:42:53.145ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "482cca3c-c919-4761-8bc3-57f695302b9c", "content-length": "170", "date": "Wed, 30 Apr 2025 06:42:52 GMT"} })
48242025-04-30T06:42:53.145ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/1/component/stage0/caboose?firmware_slot=0
48252025-04-30T06:42:53.149ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "7aa7a398-6366-4f0d-94ea-8ab4da954267", "content-length": "179", "date": "Wed, 30 Apr 2025 06:42:52 GMT"} })
48262025-04-30T06:42:53.151ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/1/component/stage0/caboose?firmware_slot=1
48272025-04-30T06:42:53.153ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "f26cb6d9-9354-4d37-b62a-0418dc66bb50", "content-length": "179", "date": "Wed, 30 Apr 2025 06:42:52 GMT"} })
48282025-04-30T06:42:53.153ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/1/component/rot/cmpa
48292025-04-30T06:42:53.155ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "9f1255eb-a094-475f-8837-4c05f82ad4ad", "content-length": "702", "date": "Wed, 30 Apr 2025 06:42:52 GMT"} })
48302025-04-30T06:42:53.155ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/1/component/rot/cfpa
48312025-04-30T06:42:53.157ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "748de990-b19d-4794-ab51-0a546efed813", "content-length": "727", "date": "Wed, 30 Apr 2025 06:42:52 GMT"} })
48322025-04-30T06:42:53.157ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/1/component/rot/cfpa
48332025-04-30T06:42:53.159ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "844e6b9e-886f-454f-ab6f-453e8372701f", "content-length": "729", "date": "Wed, 30 Apr 2025 06:42:52 GMT"} })
48342025-04-30T06:42:53.159ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/1/component/rot/cfpa
48352025-04-30T06:42:53.162ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "56b249fe-3876-482c-baca-b81316a121d8", "content-length": "728", "date": "Wed, 30 Apr 2025 06:42:52 GMT"} })
48362025-04-30T06:42:53.162ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/0
48372025-04-30T06:42:53.164ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "44cea21a-0c38-4f11-92ad-058083218b2a", "content-length": "723", "date": "Wed, 30 Apr 2025 06:42:52 GMT"} })
48382025-04-30T06:42:53.164ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/0/component/sp/caboose?firmware_slot=0
48392025-04-30T06:42:53.166ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "fa970353-50a0-43b6-9d22-39b263c6cf48", "content-length": "109", "date": "Wed, 30 Apr 2025 06:42:52 GMT"} })
48402025-04-30T06:42:53.167ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/0/component/sp/caboose?firmware_slot=1
48412025-04-30T06:42:53.172ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "2a9e0ce2-37a2-4e5a-bd5d-3a199b42ab4e", "content-length": "109", "date": "Wed, 30 Apr 2025 06:42:52 GMT"} })
48422025-04-30T06:42:53.172ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/0/component/rot/caboose?firmware_slot=0
48432025-04-30T06:42:53.176ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "ad7fcf89-d329-4c63-81a4-6f15641e04d1", "content-length": "169", "date": "Wed, 30 Apr 2025 06:42:52 GMT"} })
48442025-04-30T06:42:53.176ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/0/component/rot/caboose?firmware_slot=1
48452025-04-30T06:42:53.178ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "23714fef-0f26-4014-8448-bccaca19770f", "content-length": "169", "date": "Wed, 30 Apr 2025 06:42:52 GMT"} })
48462025-04-30T06:42:53.179ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/0/component/stage0/caboose?firmware_slot=0
48472025-04-30T06:42:53.182ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "21cbdfb1-4b6c-46dd-89d2-01d93fb102dd", "content-length": "178", "date": "Wed, 30 Apr 2025 06:42:52 GMT"} })
48482025-04-30T06:42:53.183ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/0/component/stage0/caboose?firmware_slot=1
48492025-04-30T06:42:53.191ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "7d139159-311e-48a1-a7a2-808d067f6189", "content-length": "178", "date": "Wed, 30 Apr 2025 06:42:52 GMT"} })
48502025-04-30T06:42:53.191ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/0/component/rot/cmpa
48512025-04-30T06:42:53.192ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
48522025-04-30T06:42:53.192ZTRACtest_snapshot_basic (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
48532025-04-30T06:42:53.193ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
48542025-04-30T06:42:53.193ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {} }
48552025-04-30T06:42:53.193ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
48562025-04-30T06:42:53.194ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "858e3641-6165-4fea-a3f0-195e69e7ff11", "content-length": "702", "date": "Wed, 30 Apr 2025 06:42:52 GMT"} })
48572025-04-30T06:42:53.194ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/0/component/rot/cfpa
48582025-04-30T06:42:53.202ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
48592025-04-30T06:42:53.209ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
48602025-04-30T06:42:53.210ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
48612025-04-30T06:42:53.210ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {} }
48622025-04-30T06:42:53.210ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
48632025-04-30T06:42:53.211ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "03708bb4-4835-4bd1-922d-ea60fa001576", "content-length": "727", "date": "Wed, 30 Apr 2025 06:42:53 GMT"} })
48642025-04-30T06:42:53.211ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/0/component/rot/cfpa
48652025-04-30T06:42:53.213ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "aa7e7c27-0904-4bfe-b82c-7cdd71b6333b", "content-length": "729", "date": "Wed, 30 Apr 2025 06:42:53 GMT"} })
48662025-04-30T06:42:53.213ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/0/component/rot/cfpa
48672025-04-30T06:42:53.214ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
48682025-04-30T06:42:53.214ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {} }
48692025-04-30T06:42:53.216ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
48702025-04-30T06:42:53.218ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "82ec0802-e5fe-4b45-a419-1f1955bcf32f", "content-length": "728", "date": "Wed, 30 Apr 2025 06:42:53 GMT"} })
48712025-04-30T06:42:53.218ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/1
48722025-04-30T06:42:53.232ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "7843c51f-7881-455f-97db-bdcffc7036cf", "content-length": "723", "date": "Wed, 30 Apr 2025 06:42:53 GMT"} })
48732025-04-30T06:42:53.233ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/1/component/sp/caboose?firmware_slot=0
48742025-04-30T06:42:53.236ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "fa177d12-65f7-4863-8137-7348a1a6b5f7", "content-length": "109", "date": "Wed, 30 Apr 2025 06:42:53 GMT"} })
48752025-04-30T06:42:53.236ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/1/component/sp/caboose?firmware_slot=1
48762025-04-30T06:42:53.238ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "a0c7923b-1a0b-41a2-b2a9-264aedd0ffce", "content-length": "109", "date": "Wed, 30 Apr 2025 06:42:53 GMT"} })
48772025-04-30T06:42:53.238ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/1/component/rot/caboose?firmware_slot=0
48782025-04-30T06:42:53.241ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "82adf0c0-ad93-4190-bb1c-003eea99c5ca", "content-length": "169", "date": "Wed, 30 Apr 2025 06:42:53 GMT"} })
48792025-04-30T06:42:53.241ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/1/component/rot/caboose?firmware_slot=1
48802025-04-30T06:42:53.243ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "417a58ce-b2d6-4f0c-9b79-4b81a818451c", "content-length": "169", "date": "Wed, 30 Apr 2025 06:42:53 GMT"} })
48812025-04-30T06:42:53.243ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/1/component/stage0/caboose?firmware_slot=0
48822025-04-30T06:42:53.246ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "27478725-3476-4534-ab17-b5a7c6b25495", "content-length": "178", "date": "Wed, 30 Apr 2025 06:42:53 GMT"} })
48832025-04-30T06:42:53.246ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/1/component/stage0/caboose?firmware_slot=1
48842025-04-30T06:42:53.250ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "dfe66497-a096-4629-8025-f4488342ac1f", "content-length": "178", "date": "Wed, 30 Apr 2025 06:42:53 GMT"} })
48852025-04-30T06:42:53.250ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/1/component/rot/cmpa
48862025-04-30T06:42:53.251ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "ddf2adff-1406-4262-a318-dbc0384ebc38", "content-length": "702", "date": "Wed, 30 Apr 2025 06:42:53 GMT"} })
48872025-04-30T06:42:53.251ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/1/component/rot/cfpa
48882025-04-30T06:42:53.255ZDEBGtest_snapshot_basic: attempt to fetch latest inventory collection result = Ok("found")
48892025-04-30T06:42:53.256ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "ff258466-ed57-4761-881d-aae9ee397b0e", "content-length": "727", "date": "Wed, 30 Apr 2025 06:42:53 GMT"} })
48902025-04-30T06:42:53.256ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/1/component/rot/cfpa
48912025-04-30T06:42:53.259ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "291cb640-73ad-4ab0-ab9e-44a9483eecc8", "content-length": "729", "date": "Wed, 30 Apr 2025 06:42:53 GMT"} })
48922025-04-30T06:42:53.259ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/1/component/rot/cfpa
48932025-04-30T06:42:53.260ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "9c0221c9-b899-4659-bc4c-9ba3b75ab4a9", "content-length": "728", "date": "Wed, 30 Apr 2025 06:42:53 GMT"} })
48942025-04-30T06:42:53.260ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
48952025-04-30T06:42:53.260ZTRACtest_snapshot_basic (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
48962025-04-30T06:42:53.260ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
48972025-04-30T06:42:53.260ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
48982025-04-30T06:42:53.261ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
48992025-04-30T06:42:53.309ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
49002025-04-30T06:42:53.311ZTRACtest_snapshot_basic (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
49012025-04-30T06:42:53.311ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
49022025-04-30T06:42:53.311ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {} }
49032025-04-30T06:42:53.313ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
49042025-04-30T06:42:53.314ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
49052025-04-30T06:42:53.325ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
49062025-04-30T06:42:53.325ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
49072025-04-30T06:42:53.325ZTRACtest_snapshot_basic (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
49082025-04-30T06:42:53.325ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
49092025-04-30T06:42:53.325ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
49102025-04-30T06:42:53.325ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
49112025-04-30T06:42:53.330ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
49122025-04-30T06:42:53.341ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
49132025-04-30T06:42:53.341ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
49142025-04-30T06:42:53.341ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {} }
49152025-04-30T06:42:53.350ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
49162025-04-30T06:42:53.361ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
49172025-04-30T06:42:53.366ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
49182025-04-30T06:42:53.366ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
49192025-04-30T06:42:53.366ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
49202025-04-30T06:42:53.366ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
49212025-04-30T06:42:53.368ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
49222025-04-30T06:42:53.369ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {} }
49232025-04-30T06:42:53.369ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
49242025-04-30T06:42:53.595ZDEBGtest_snapshot_basic (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:41923
49252025-04-30T06:42:53.596ZDEBGtest_snapshot_basic (BackgroundTasks): client request SledAgent = http://[::1]:41923 background_task = inventory_collection body = None method = GET uri = http://[::1]:41923/inventory
49262025-04-30T06:42:53.598ZINFOtest_snapshot_basic (dropshot): accepted connection local_addr = [::1]:41923 remote_addr = [::1]:60239 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
49272025-04-30T06:42:53.601ZTRACtest_snapshot_basic (dropshot): incoming request local_addr = [::1]:41923 method = GET remote_addr = [::1]:60239 req_id = b327ce7f-e9a7-4949-a313-111166410926 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
49282025-04-30T06:42:53.602ZINFOtest_snapshot_basic (dropshot): request completed latency_us = 674 local_addr = [::1]:41923 method = GET remote_addr = [::1]:60239 req_id = b327ce7f-e9a7-4949-a313-111166410926 response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
49292025-04-30T06:42:53.604ZDEBGtest_snapshot_basic (BackgroundTasks): client response SledAgent = http://[::1]:41923 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:41923/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "b327ce7f-e9a7-4949-a313-111166410926", "content-length": "3141", "date": "Wed, 30 Apr 2025 06:42:53 GMT"} })
49302025-04-30T06:42:53.607ZDEBGtest_snapshot_basic (BackgroundTasks): begin collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
49312025-04-30T06:42:53.607ZDEBGtest_snapshot_basic (BackgroundTasks): end collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
49322025-04-30T06:42:53.607ZDEBGtest_snapshot_basic (BackgroundTasks): finished collection background_task = inventory_collection
49332025-04-30T06:42:53.610ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
49342025-04-30T06:42:53.610ZTRACtest_snapshot_basic (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
49352025-04-30T06:42:53.610ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
49362025-04-30T06:42:53.611ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
49372025-04-30T06:42:53.616ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
49382025-04-30T06:42:53.645ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
49392025-04-30T06:42:53.650ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
49402025-04-30T06:42:53.689ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
49412025-04-30T06:42:53.689ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
49422025-04-30T06:42:53.690ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
49432025-04-30T06:42:53.725ZINFOtest_snapshot_basic (oximeter-agent): refreshing list of producers from Nexus collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 file = oximeter/collector/src/agent.rs:562
49442025-04-30T06:42:53.725ZDEBGtest_snapshot_basic (oximeter): client request body = None method = GET uri = http://127.0.0.1:61468/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
49452025-04-30T06:42:53.726ZTRACtest_snapshot_basic (dropshot_internal): incoming request local_addr = 127.0.0.1:61468 method = GET remote_addr = 127.0.0.1:64332 req_id = 58acbc5c-f9b6-42bc-bcdc-219d40fa2cb0 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
49462025-04-30T06:42:53.728ZTRACtest_snapshot_basic (dropshot_internal): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:61468 method = GET remote_addr = 127.0.0.1:64332 req_id = 58acbc5c-f9b6-42bc-bcdc-219d40fa2cb0 resource = Database uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
49472025-04-30T06:42:53.728ZDEBGtest_snapshot_basic (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:61468 method = GET remote_addr = 127.0.0.1:64332 req_id = 58acbc5c-f9b6-42bc-bcdc-219d40fa2cb0 roles = RoleSet { roles: {} } uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
49482025-04-30T06:42:53.729ZDEBGtest_snapshot_basic (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:61468 method = GET remote_addr = 127.0.0.1:64332 req_id = 58acbc5c-f9b6-42bc-bcdc-219d40fa2cb0 resource = Database result = Ok(()) uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
49492025-04-30T06:42:53.744ZINFOtest_snapshot_basic (dropshot_internal): request completed latency_us = 17696 local_addr = 127.0.0.1:61468 method = GET remote_addr = 127.0.0.1:64332 req_id = 58acbc5c-f9b6-42bc-bcdc-219d40fa2cb0 response_code = 200 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
49502025-04-30T06:42:53.744ZDEBGtest_snapshot_basic (oximeter): client response result = Ok(Response { url: "http://127.0.0.1:61468/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending", status: 200, headers: {"content-type": "application/json", "x-request-id": "58acbc5c-f9b6-42bc-bcdc-219d40fa2cb0", "content-length": "405", "date": "Wed, 30 Apr 2025 06:42:53 GMT"} })
49512025-04-30T06:42:53.745ZDEBGtest_snapshot_basic (oximeter): client request body = None method = GET uri = http://127.0.0.1:61468/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
49522025-04-30T06:42:53.745ZTRACtest_snapshot_basic (dropshot_internal): incoming request local_addr = 127.0.0.1:61468 method = GET remote_addr = 127.0.0.1:64332 req_id = c3851446-cce4-407a-a870-0f08cfa7573a uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
49532025-04-30T06:42:53.753ZTRACtest_snapshot_basic (dropshot_internal): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:61468 method = GET remote_addr = 127.0.0.1:64332 req_id = c3851446-cce4-407a-a870-0f08cfa7573a resource = Database uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
49542025-04-30T06:42:53.753ZDEBGtest_snapshot_basic (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:61468 method = GET remote_addr = 127.0.0.1:64332 req_id = c3851446-cce4-407a-a870-0f08cfa7573a roles = RoleSet { roles: {} } uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
49552025-04-30T06:42:53.754ZDEBGtest_snapshot_basic (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:61468 method = GET remote_addr = 127.0.0.1:64332 req_id = c3851446-cce4-407a-a870-0f08cfa7573a resource = Database result = Ok(()) uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
49562025-04-30T06:42:53.785ZDEBGtest_snapshot_basic: attempt to fetch latest inventory collection result = Ok("found")
49572025-04-30T06:42:53.789ZINFOtest_snapshot_basic (dropshot_internal): request completed latency_us = 44021 local_addr = 127.0.0.1:61468 method = GET remote_addr = 127.0.0.1:64332 req_id = c3851446-cce4-407a-a870-0f08cfa7573a response_code = 200 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
49582025-04-30T06:42:53.793ZDEBGtest_snapshot_basic (oximeter): client response result = Ok(Response { url: "http://127.0.0.1:61468/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D", status: 200, headers: {"content-type": "application/json", "x-request-id": "c3851446-cce4-407a-a870-0f08cfa7573a", "content-length": "29", "date": "Wed, 30 Apr 2025 06:42:53 GMT"} })
49592025-04-30T06:42:53.793ZTRACtest_snapshot_basic (oximeter-agent): ignoring request to update existing metric producer, since the endpoint information is the same as the existing collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = 354a52d4-3078-4706-81ce-12c32f6bbeaa
49602025-04-30T06:42:53.793ZTRACtest_snapshot_basic (oximeter-agent): ignoring request to update existing metric producer, since the endpoint information is the same as the existing collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
49612025-04-30T06:42:53.793ZINFOtest_snapshot_basic (oximeter-agent): refreshed list of producers from Nexus collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 file = oximeter/collector/src/agent.rs:637 n_current_tasks = 2 n_pruned_tasks = 0
49622025-04-30T06:42:53.794ZWARNtest_snapshot_basic (datastore): Non-retryable transaction failure err = DatabaseError(UniqueViolation, "duplicate key value violates unique constraint \\"inv_caboose_pkey\\"") transaction = inventory_insert_collection
49632025-04-30T06:42:53.796ZWARNtest_snapshot_basic (BackgroundTasks): inventory collection failed background_task = inventory_collection error = failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint "inv_caboose_pkey"
49642025-04-30T06:42:53.796ZDEBGtest_snapshot_basic (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 1.090869715s iteration = 11 status = {"error":"failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint \\"inv_caboose_pkey\\""}
49652025-04-30T06:42:53.796ZDEBGtest_snapshot_basic (BackgroundTasks): activating background_task = inventory_collection iteration = 12 reason = Signaled
49662025-04-30T06:42:53.796ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
49672025-04-30T06:42:53.796ZTRACtest_snapshot_basic (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
49682025-04-30T06:42:53.796ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
49692025-04-30T06:42:53.797ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
49702025-04-30T06:42:53.797ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
49712025-04-30T06:42:53.800ZDEBGtest_snapshot_basic (collection-task): sent timer-based collection request to the collection task collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
49722025-04-30T06:42:53.800ZDEBGtest_snapshot_basic (collection-task): collecting from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
49732025-04-30T06:42:53.801ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
49742025-04-30T06:42:53.807ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
49752025-04-30T06:42:53.807ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
49762025-04-30T06:42:53.807ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
49772025-04-30T06:42:53.808ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
49782025-04-30T06:42:53.816ZTRACtest_snapshot_basic (collection-task): sent collection request to producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
49792025-04-30T06:42:53.817ZDEBGtest_snapshot_basic (collection-task): collected results from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 n_results = 1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
49802025-04-30T06:42:53.817ZTRACtest_snapshot_basic (collection-task): forwarded results to main collection loop collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
49812025-04-30T06:42:53.817ZTRACtest_snapshot_basic (collection-task): top of inner collection loop, waiting for next request collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
49822025-04-30T06:42:53.818ZDEBGtest_snapshot_basic (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = [(d1ad44e5-18dd-4193-a70e-bf9042f0e0e1, 0)]
49832025-04-30T06:42:53.818ZTRACtest_snapshot_basic (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
49842025-04-30T06:42:53.818ZTRACtest_snapshot_basic (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: 4e 65 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 Ne..........._mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:53373 req_id = 7e7ff1ab-f8a1-4f1b-9e6d-f64f87342419
49852025-04-30T06:42:53.819ZDEBGtest_snapshot_basic (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 20069,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:53373 req_id = 7e7ff1ab-f8a1-4f1b-9e6d-f64f87342419
49862025-04-30T06:42:53.820ZDEBGtest_snapshot_basic (store): query key key = _mgs._tcp
49872025-04-30T06:42:53.820ZDEBGtest_snapshot_basic (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
49882025-04-30T06:42:53.820ZDEBGtest_snapshot_basic (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:53373 query = LowerQuery { name: LowerName(Name("_mgs._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 41502, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = 7e7ff1ab-f8a1-4f1b-9e6d-f64f87342419
49892025-04-30T06:42:53.821ZDEBGtest_snapshot_basic (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 41502, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 4238, tv_nsec: 620119646 } })
49902025-04-30T06:42:53.821ZTRACtest_snapshot_basic (DnsResolver): lookup_service_targets: looking up SRV target
49912025-04-30T06:42:53.821ZTRACtest_snapshot_basic (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: 85 cc 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e ............-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:64644 req_id = 264c6dff-f7b3-41df-8f93-15249076099c
49922025-04-30T06:42:53.821ZDEBGtest_snapshot_basic (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 34252,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:64644 req_id = 264c6dff-f7b3-41df-8f93-15249076099c
49932025-04-30T06:42:53.821ZDEBGtest_snapshot_basic (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
49942025-04-30T06:42:53.822ZDEBGtest_snapshot_basic (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:64644 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = 264c6dff-f7b3-41df-8f93-15249076099c
49952025-04-30T06:42:53.883ZTRACtest_snapshot_basic (DnsResolver): lookup_all_socket_v6 srv dns_name = _clickhouse-admin-keeper._tcp.control-plane.oxide.internal
49962025-04-30T06:42:53.883ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
49972025-04-30T06:42:53.883ZTRACtest_snapshot_basic (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
49982025-04-30T06:42:53.883ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
49992025-04-30T06:42:53.883ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {} }
50002025-04-30T06:42:53.884ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
50012025-04-30T06:42:53.884ZTRACtest_snapshot_basic (dns): buffer buffer = Length: 87 (0x57) bytes\n0000: 8e 49 01 00 00 01 00 00 00 00 00 01 18 5f 63 6c .I..........._cl\n0010: 69 63 6b 68 6f 75 73 65 2d 61 64 6d 69 6e 2d 6b ickhouse-admin-k\n0020: 65 65 70 65 72 04 5f 74 63 70 0d 63 6f 6e 74 72 eeper._tcp.contr\n0030: 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 69 ol-plane.oxide.i\n0040: 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 04 nternal..!....).\n0050: d0 00 00 00 00 00 00 ....... kind = dns peer_addr = [::1]:59659 req_id = 5a9fb688-9b99-4ae0-9715-0407174a7d24
50022025-04-30T06:42:53.884ZDEBGtest_snapshot_basic (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 36425,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 24,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 97,\n 100,\n 109,\n 105,\n 110,\n 45,\n 107,\n 101,\n 101,\n 112,\n 101,\n 114,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:59659 req_id = 5a9fb688-9b99-4ae0-9715-0407174a7d24
50032025-04-30T06:42:53.885ZDEBGtest_snapshot_basic (store): query key key = _clickhouse-admin-keeper._tcp
50042025-04-30T06:42:53.885ZERROtest_snapshot_basic (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 36425, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } NXDOMAIN: no records found for name: "_clickhouse-admin-keeper._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:59659 req_id = 5a9fb688-9b99-4ae0-9715-0407174a7d24
50052025-04-30T06:42:53.885ZDEBGtest_snapshot_basic (BackgroundTasks): begin collection background_task = inventory_collection
50062025-04-30T06:42:53.887ZDEBGtest_snapshot_basic (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:41502
50072025-04-30T06:42:53.887ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/ignition
50082025-04-30T06:42:53.888ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
50092025-04-30T06:42:53.893ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
50102025-04-30T06:42:53.893ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
50112025-04-30T06:42:53.893ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {} }
50122025-04-30T06:42:53.893ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
50132025-04-30T06:42:53.895ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "cf3d01d5-c691-499c-8ed7-11172a7a8cf4", "content-length": "835", "date": "Wed, 30 Apr 2025 06:42:53 GMT"} })
50142025-04-30T06:42:53.896ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/0
50152025-04-30T06:42:53.897ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
50162025-04-30T06:42:53.897ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {} }
50172025-04-30T06:42:53.898ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
50182025-04-30T06:42:53.899ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "68930299-cf63-4e50-976b-ae32c0161538", "content-length": "734", "date": "Wed, 30 Apr 2025 06:42:53 GMT"} })
50192025-04-30T06:42:53.899ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/0/component/sp/caboose?firmware_slot=0
50202025-04-30T06:42:53.906ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "6f238f10-5e03-4d19-b7a7-8e5146306188", "content-length": "111", "date": "Wed, 30 Apr 2025 06:42:53 GMT"} })
50212025-04-30T06:42:53.906ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/0/component/sp/caboose?firmware_slot=1
50222025-04-30T06:42:53.910ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "cc8b0e75-8593-4ec8-a787-f5b6c80903aa", "content-length": "111", "date": "Wed, 30 Apr 2025 06:42:53 GMT"} })
50232025-04-30T06:42:53.911ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/0/component/rot/caboose?firmware_slot=0
50242025-04-30T06:42:53.914ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "5ac8235b-2a09-445b-989e-74cd33aa34fd", "content-length": "170", "date": "Wed, 30 Apr 2025 06:42:53 GMT"} })
50252025-04-30T06:42:53.915ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/0/component/rot/caboose?firmware_slot=1
50262025-04-30T06:42:53.921ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "4ea5bee2-90ea-4c5d-bb16-f94d9ce7aeae", "content-length": "170", "date": "Wed, 30 Apr 2025 06:42:53 GMT"} })
50272025-04-30T06:42:53.922ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/0/component/stage0/caboose?firmware_slot=0
50282025-04-30T06:42:53.926ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "3f437da4-3585-4ba3-914a-27576a729a34", "content-length": "179", "date": "Wed, 30 Apr 2025 06:42:53 GMT"} })
50292025-04-30T06:42:53.926ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/0/component/stage0/caboose?firmware_slot=1
50302025-04-30T06:42:53.929ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "c27f54ac-2ce8-4938-918f-b871d7ec1889", "content-length": "179", "date": "Wed, 30 Apr 2025 06:42:53 GMT"} })
50312025-04-30T06:42:53.929ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/0/component/rot/cmpa
50322025-04-30T06:42:53.935ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "522d34bd-38c7-460a-8394-ce5231fe97bf", "content-length": "702", "date": "Wed, 30 Apr 2025 06:42:53 GMT"} })
50332025-04-30T06:42:53.938ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/0/component/rot/cfpa
50342025-04-30T06:42:53.938ZDEBGtest_snapshot_basic: attempt to fetch latest inventory collection result = Ok("found")
50352025-04-30T06:42:53.942ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "a0ef0767-85e5-4460-82fb-7a8df1d34e95", "content-length": "727", "date": "Wed, 30 Apr 2025 06:42:53 GMT"} })
50362025-04-30T06:42:53.942ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/0/component/rot/cfpa
50372025-04-30T06:42:53.945ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "2df549c9-f28f-42a6-ab1c-3235c68ec859", "content-length": "729", "date": "Wed, 30 Apr 2025 06:42:53 GMT"} })
50382025-04-30T06:42:53.945ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/0/component/rot/cfpa
50392025-04-30T06:42:53.947ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "25db48f8-dade-4d5c-a425-45cbbb9d6d2d", "content-length": "728", "date": "Wed, 30 Apr 2025 06:42:53 GMT"} })
50402025-04-30T06:42:53.948ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/1
50412025-04-30T06:42:53.951ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "9ebdaa60-ca34-4075-871a-f586cbb69350", "content-length": "734", "date": "Wed, 30 Apr 2025 06:42:53 GMT"} })
50422025-04-30T06:42:53.952ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/1/component/sp/caboose?firmware_slot=0
50432025-04-30T06:42:53.965ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "049fb699-0909-4fb3-a252-e401762931b8", "content-length": "111", "date": "Wed, 30 Apr 2025 06:42:53 GMT"} })
50442025-04-30T06:42:53.965ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/1/component/sp/caboose?firmware_slot=1
50452025-04-30T06:42:53.969ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "44bc7622-0f98-49b4-b419-4bd837b7ca98", "content-length": "111", "date": "Wed, 30 Apr 2025 06:42:53 GMT"} })
50462025-04-30T06:42:53.969ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/1/component/rot/caboose?firmware_slot=0
50472025-04-30T06:42:53.971ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "7e005d2c-8c68-4056-8538-38ccce45dc35", "content-length": "170", "date": "Wed, 30 Apr 2025 06:42:53 GMT"} })
50482025-04-30T06:42:53.971ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/1/component/rot/caboose?firmware_slot=1
50492025-04-30T06:42:53.973ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "523c9d64-dc1b-465d-a878-711e94d5d5c8", "content-length": "170", "date": "Wed, 30 Apr 2025 06:42:53 GMT"} })
50502025-04-30T06:42:53.973ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/1/component/stage0/caboose?firmware_slot=0
50512025-04-30T06:42:53.975ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "89861eb7-0546-4498-a75a-ba6a48a56175", "content-length": "179", "date": "Wed, 30 Apr 2025 06:42:53 GMT"} })
50522025-04-30T06:42:53.975ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/1/component/stage0/caboose?firmware_slot=1
50532025-04-30T06:42:53.977ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "91784c72-1c68-4eb2-ac6a-5452b47997ab", "content-length": "179", "date": "Wed, 30 Apr 2025 06:42:53 GMT"} })
50542025-04-30T06:42:53.977ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/1/component/rot/cmpa
50552025-04-30T06:42:53.978ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "1097a649-b115-4f05-9134-383f59083f8e", "content-length": "702", "date": "Wed, 30 Apr 2025 06:42:53 GMT"} })
50562025-04-30T06:42:53.978ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/1/component/rot/cfpa
50572025-04-30T06:42:53.981ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "44e721d5-8f54-4f28-b784-bc9d334a3544", "content-length": "727", "date": "Wed, 30 Apr 2025 06:42:53 GMT"} })
50582025-04-30T06:42:53.981ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/1/component/rot/cfpa
50592025-04-30T06:42:53.982ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "64f1e8f4-e4b3-4397-9c22-8a04c672f3e8", "content-length": "729", "date": "Wed, 30 Apr 2025 06:42:53 GMT"} })
50602025-04-30T06:42:53.982ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/1/component/rot/cfpa
50612025-04-30T06:42:53.984ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "645bfa23-20ec-4163-a604-3dd18208226d", "content-length": "728", "date": "Wed, 30 Apr 2025 06:42:53 GMT"} })
50622025-04-30T06:42:53.984ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/0
50632025-04-30T06:42:53.988ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "4181ec1c-29ef-4d8a-91f5-865324810091", "content-length": "723", "date": "Wed, 30 Apr 2025 06:42:53 GMT"} })
50642025-04-30T06:42:53.990ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/0/component/sp/caboose?firmware_slot=0
50652025-04-30T06:42:53.990ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
50662025-04-30T06:42:53.990ZTRACtest_snapshot_basic (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
50672025-04-30T06:42:53.990ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
50682025-04-30T06:42:53.990ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {} }
50692025-04-30T06:42:53.992ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
50702025-04-30T06:42:53.996ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "a9bb0d78-6337-4007-a388-bf9c49273471", "content-length": "109", "date": "Wed, 30 Apr 2025 06:42:53 GMT"} })
50712025-04-30T06:42:53.998ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/0/component/sp/caboose?firmware_slot=1
50722025-04-30T06:42:53.998ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
50732025-04-30T06:42:54.008ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
50742025-04-30T06:42:54.008ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
50752025-04-30T06:42:54.008ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {} }
50762025-04-30T06:42:54.009ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
50772025-04-30T06:42:54.013ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
50782025-04-30T06:42:54.013ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {} }
50792025-04-30T06:42:54.016ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
50802025-04-30T06:42:54.017ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "bce77ab8-f695-48c9-b49e-56a551c30d71", "content-length": "109", "date": "Wed, 30 Apr 2025 06:42:53 GMT"} })
50812025-04-30T06:42:54.017ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/0/component/rot/caboose?firmware_slot=0
50822025-04-30T06:42:54.022ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "8436e221-0911-4dee-ba9a-1a4b8c246a4e", "content-length": "169", "date": "Wed, 30 Apr 2025 06:42:53 GMT"} })
50832025-04-30T06:42:54.022ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/0/component/rot/caboose?firmware_slot=1
50842025-04-30T06:42:54.026ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "86c629b2-5d62-4957-95fa-766c675cd010", "content-length": "169", "date": "Wed, 30 Apr 2025 06:42:53 GMT"} })
50852025-04-30T06:42:54.026ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/0/component/stage0/caboose?firmware_slot=0
50862025-04-30T06:42:54.032ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "ea29785e-c000-4d52-8d0a-73091fd9a2bc", "content-length": "178", "date": "Wed, 30 Apr 2025 06:42:53 GMT"} })
50872025-04-30T06:42:54.033ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/0/component/stage0/caboose?firmware_slot=1
50882025-04-30T06:42:54.035ZDEBGtest_snapshot_basic (results-sink): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
50892025-04-30T06:42:54.035ZDEBGtest_snapshot_basic (results-sink-cluster): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
50902025-04-30T06:42:54.035ZDEBGtest_snapshot_basic (clickhouse-client): unrolling 1 total samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 6ad06c6b-9a6e-4ed1-b3f4-5d2e9da3f8a7
50912025-04-30T06:42:54.036ZTRACtest_snapshot_basic (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 6ad06c6b-9a6e-4ed1-b3f4-5d2e9da3f8a7 n_columns = 4 n_rows = 2 sql = INSERT INTO oximeter.fields_string FORMAT Native
50922025-04-30T06:42:54.044ZDEBGtest_snapshot_basic (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 6ad06c6b-9a6e-4ed1-b3f4-5d2e9da3f8a7 n_rows = 2 table_name = fields_string
50932025-04-30T06:42:54.044ZTRACtest_snapshot_basic (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 6ad06c6b-9a6e-4ed1-b3f4-5d2e9da3f8a7 n_columns = 4 n_rows = 1 sql = INSERT INTO oximeter.measurements_i64 FORMAT Native
50942025-04-30T06:42:54.044ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "18a886e0-7f63-494c-b648-5ac90de7abbf", "content-length": "178", "date": "Wed, 30 Apr 2025 06:42:53 GMT"} })
50952025-04-30T06:42:54.045ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/0/component/rot/cmpa
50962025-04-30T06:42:54.047ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "20e5620a-cebf-4a82-8fb1-c9e5e7228b01", "content-length": "702", "date": "Wed, 30 Apr 2025 06:42:53 GMT"} })
50972025-04-30T06:42:54.047ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/0/component/rot/cfpa
50982025-04-30T06:42:54.048ZDEBGtest_snapshot_basic (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 6ad06c6b-9a6e-4ed1-b3f4-5d2e9da3f8a7 n_rows = 1 table_name = measurements_i64
50992025-04-30T06:42:54.049ZTRACtest_snapshot_basic (results-sink): successfully inserted samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
51002025-04-30T06:42:54.051ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "479ff885-4988-4d31-9620-0f09a277d11d", "content-length": "727", "date": "Wed, 30 Apr 2025 06:42:53 GMT"} })
51012025-04-30T06:42:54.052ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/0/component/rot/cfpa
51022025-04-30T06:42:54.058ZDEBGtest_snapshot_basic: attempt to fetch latest inventory collection result = Ok("found")
51032025-04-30T06:42:54.061ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "58aea82f-ecaa-495a-9457-da2afeb97102", "content-length": "729", "date": "Wed, 30 Apr 2025 06:42:53 GMT"} })
51042025-04-30T06:42:54.061ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/0/component/rot/cfpa
51052025-04-30T06:42:54.062ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "90f24237-3fab-4e78-86d1-9a013f63aace", "content-length": "728", "date": "Wed, 30 Apr 2025 06:42:53 GMT"} })
51062025-04-30T06:42:54.062ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/1
51072025-04-30T06:42:54.064ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "a66d0da9-8a29-4262-b5d8-6e2a6be22ca1", "content-length": "723", "date": "Wed, 30 Apr 2025 06:42:53 GMT"} })
51082025-04-30T06:42:54.064ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/1/component/sp/caboose?firmware_slot=0
51092025-04-30T06:42:54.067ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "48e13045-ed7d-4bb0-bd89-b0f765dc3063", "content-length": "109", "date": "Wed, 30 Apr 2025 06:42:53 GMT"} })
51102025-04-30T06:42:54.067ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/1/component/sp/caboose?firmware_slot=1
51112025-04-30T06:42:54.069ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "72e193a6-f0b8-4d76-8853-055e206955df", "content-length": "109", "date": "Wed, 30 Apr 2025 06:42:53 GMT"} })
51122025-04-30T06:42:54.069ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/1/component/rot/caboose?firmware_slot=0
51132025-04-30T06:42:54.073ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "e204af26-dd09-4df1-b137-c0c136257cab", "content-length": "169", "date": "Wed, 30 Apr 2025 06:42:53 GMT"} })
51142025-04-30T06:42:54.073ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/1/component/rot/caboose?firmware_slot=1
51152025-04-30T06:42:54.075ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "af9fef37-040e-4eea-aee2-8c953ec3c71a", "content-length": "169", "date": "Wed, 30 Apr 2025 06:42:53 GMT"} })
51162025-04-30T06:42:54.075ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/1/component/stage0/caboose?firmware_slot=0
51172025-04-30T06:42:54.077ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "4859c3cb-d972-49cf-b75f-2e81af3120fb", "content-length": "178", "date": "Wed, 30 Apr 2025 06:42:53 GMT"} })
51182025-04-30T06:42:54.077ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/1/component/stage0/caboose?firmware_slot=1
51192025-04-30T06:42:54.080ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "da991751-7289-4a6d-bb45-2a1bf9a44774", "content-length": "178", "date": "Wed, 30 Apr 2025 06:42:53 GMT"} })
51202025-04-30T06:42:54.080ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/1/component/rot/cmpa
51212025-04-30T06:42:54.081ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "41a0f906-46f7-458d-9464-e68e343b0285", "content-length": "702", "date": "Wed, 30 Apr 2025 06:42:53 GMT"} })
51222025-04-30T06:42:54.081ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/1/component/rot/cfpa
51232025-04-30T06:42:54.082ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "692ae202-b7d5-4a44-a74e-d49042c42d49", "content-length": "727", "date": "Wed, 30 Apr 2025 06:42:53 GMT"} })
51242025-04-30T06:42:54.082ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/1/component/rot/cfpa
51252025-04-30T06:42:54.083ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "9079133a-d6ac-4403-9030-e5c0b6c20e7c", "content-length": "729", "date": "Wed, 30 Apr 2025 06:42:53 GMT"} })
51262025-04-30T06:42:54.083ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/1/component/rot/cfpa
51272025-04-30T06:42:54.084ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "82b6ffee-4478-432d-b186-a67fb0775e62", "content-length": "728", "date": "Wed, 30 Apr 2025 06:42:53 GMT"} })
51282025-04-30T06:42:54.084ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
51292025-04-30T06:42:54.084ZTRACtest_snapshot_basic (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
51302025-04-30T06:42:54.084ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
51312025-04-30T06:42:54.084ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
51322025-04-30T06:42:54.085ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
51332025-04-30T06:42:54.087ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
51342025-04-30T06:42:54.092ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
51352025-04-30T06:42:54.092ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
51362025-04-30T06:42:54.092ZTRACtest_snapshot_basic (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
51372025-04-30T06:42:54.092ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
51382025-04-30T06:42:54.092ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
51392025-04-30T06:42:54.092ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
51402025-04-30T06:42:54.096ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
51412025-04-30T06:42:54.101ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
51422025-04-30T06:42:54.101ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
51432025-04-30T06:42:54.101ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
51442025-04-30T06:42:54.101ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
51452025-04-30T06:42:54.157ZDEBGtest_snapshot_basic (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:41923
51462025-04-30T06:42:54.157ZDEBGtest_snapshot_basic (BackgroundTasks): client request SledAgent = http://[::1]:41923 background_task = inventory_collection body = None method = GET uri = http://[::1]:41923/inventory
51472025-04-30T06:42:54.158ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
51482025-04-30T06:42:54.158ZTRACtest_snapshot_basic (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
51492025-04-30T06:42:54.158ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
51502025-04-30T06:42:54.158ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {} }
51512025-04-30T06:42:54.159ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
51522025-04-30T06:42:54.159ZINFOtest_snapshot_basic (dropshot): accepted connection local_addr = [::1]:41923 remote_addr = [::1]:49884 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
51532025-04-30T06:42:54.160ZWARNtest_snapshot_basic (results-sink-cluster): failed to insert some results into metric DB collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 error = Failed to check out connection to database: No backends found for this service file = oximeter/collector/src/results_sink.rs:92
51542025-04-30T06:42:54.160ZTRACtest_snapshot_basic (dropshot): incoming request local_addr = [::1]:41923 method = GET remote_addr = [::1]:49884 req_id = e5ba65f9-99be-4a5b-8673-b48d3e89948d sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
51552025-04-30T06:42:54.161ZINFOtest_snapshot_basic (dropshot): request completed latency_us = 729 local_addr = [::1]:41923 method = GET remote_addr = [::1]:49884 req_id = e5ba65f9-99be-4a5b-8673-b48d3e89948d response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
51562025-04-30T06:42:54.161ZDEBGtest_snapshot_basic (BackgroundTasks): client response SledAgent = http://[::1]:41923 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:41923/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "e5ba65f9-99be-4a5b-8673-b48d3e89948d", "content-length": "3141", "date": "Wed, 30 Apr 2025 06:42:53 GMT"} })
51572025-04-30T06:42:54.164ZDEBGtest_snapshot_basic (BackgroundTasks): begin collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
51582025-04-30T06:42:54.164ZDEBGtest_snapshot_basic (BackgroundTasks): end collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
51592025-04-30T06:42:54.164ZDEBGtest_snapshot_basic (BackgroundTasks): finished collection background_task = inventory_collection
51602025-04-30T06:42:54.166ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
51612025-04-30T06:42:54.166ZTRACtest_snapshot_basic (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
51622025-04-30T06:42:54.166ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
51632025-04-30T06:42:54.166ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
51642025-04-30T06:42:54.167ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
51652025-04-30T06:42:54.168ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
51662025-04-30T06:42:54.173ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
51672025-04-30T06:42:54.173ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
51682025-04-30T06:42:54.173ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {} }
51692025-04-30T06:42:54.173ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
51702025-04-30T06:42:54.175ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
51712025-04-30T06:42:54.179ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
51722025-04-30T06:42:54.179ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
51732025-04-30T06:42:54.179ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
51742025-04-30T06:42:54.179ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
51752025-04-30T06:42:54.181ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
51762025-04-30T06:42:54.182ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {} }
51772025-04-30T06:42:54.182ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
51782025-04-30T06:42:54.212ZWARNtest_snapshot_basic (datastore): Non-retryable transaction failure err = DatabaseError(UniqueViolation, "duplicate key value violates unique constraint \\"inv_caboose_pkey\\"") transaction = inventory_insert_collection
51792025-04-30T06:42:54.212ZWARNtest_snapshot_basic (BackgroundTasks): inventory collection failed background_task = inventory_collection error = failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint "inv_caboose_pkey"
51802025-04-30T06:42:54.212ZDEBGtest_snapshot_basic (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 415.805575ms iteration = 12 status = {"error":"failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint \\"inv_caboose_pkey\\""}
51812025-04-30T06:42:54.212ZDEBGtest_snapshot_basic (BackgroundTasks): activating background_task = inventory_collection iteration = 13 reason = Signaled
51822025-04-30T06:42:54.212ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
51832025-04-30T06:42:54.212ZTRACtest_snapshot_basic (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
51842025-04-30T06:42:54.212ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
51852025-04-30T06:42:54.212ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
51862025-04-30T06:42:54.214ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
51872025-04-30T06:42:54.222ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
51882025-04-30T06:42:54.226ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
51892025-04-30T06:42:54.226ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
51902025-04-30T06:42:54.226ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
51912025-04-30T06:42:54.227ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
51922025-04-30T06:42:54.230ZDEBGtest_snapshot_basic (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = [(d1ad44e5-18dd-4193-a70e-bf9042f0e0e1, 0)]
51932025-04-30T06:42:54.231ZTRACtest_snapshot_basic (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
51942025-04-30T06:42:54.232ZDEBGtest_snapshot_basic: attempt to fetch latest inventory collection result = Ok("found")
51952025-04-30T06:42:54.233ZTRACtest_snapshot_basic (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: 54 6a 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 Tj..........._mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:58298 req_id = a1e27391-dc6d-4762-be92-6a9cad3ec645
51962025-04-30T06:42:54.233ZDEBGtest_snapshot_basic (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 21610,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:58298 req_id = a1e27391-dc6d-4762-be92-6a9cad3ec645
51972025-04-30T06:42:54.233ZDEBGtest_snapshot_basic (store): query key key = _mgs._tcp
51982025-04-30T06:42:54.233ZDEBGtest_snapshot_basic (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
51992025-04-30T06:42:54.236ZDEBGtest_snapshot_basic (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:58298 query = LowerQuery { name: LowerName(Name("_mgs._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 41502, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = a1e27391-dc6d-4762-be92-6a9cad3ec645
52002025-04-30T06:42:54.237ZDEBGtest_snapshot_basic (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 41502, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 4239, tv_nsec: 36113456 } })
52012025-04-30T06:42:54.237ZTRACtest_snapshot_basic (DnsResolver): lookup_service_targets: looking up SRV target
52022025-04-30T06:42:54.237ZTRACtest_snapshot_basic (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: b5 20 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e . ..........-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:52041 req_id = dd8e637f-3f4d-4cb7-9ce3-f3a66f233e30
52032025-04-30T06:42:54.237ZDEBGtest_snapshot_basic (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 46368,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:52041 req_id = dd8e637f-3f4d-4cb7-9ce3-f3a66f233e30
52042025-04-30T06:42:54.237ZDEBGtest_snapshot_basic (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
52052025-04-30T06:42:54.238ZDEBGtest_snapshot_basic (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:52041 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = dd8e637f-3f4d-4cb7-9ce3-f3a66f233e30
52062025-04-30T06:42:54.299ZTRACtest_snapshot_basic (DnsResolver): lookup_all_socket_v6 srv dns_name = _clickhouse-admin-keeper._tcp.control-plane.oxide.internal
52072025-04-30T06:42:54.299ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
52082025-04-30T06:42:54.301ZTRACtest_snapshot_basic (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
52092025-04-30T06:42:54.301ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
52102025-04-30T06:42:54.301ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {} }
52112025-04-30T06:42:54.302ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
52122025-04-30T06:42:54.303ZTRACtest_snapshot_basic (dns): buffer buffer = Length: 87 (0x57) bytes\n0000: 97 a9 01 00 00 01 00 00 00 00 00 01 18 5f 63 6c ............._cl\n0010: 69 63 6b 68 6f 75 73 65 2d 61 64 6d 69 6e 2d 6b ickhouse-admin-k\n0020: 65 65 70 65 72 04 5f 74 63 70 0d 63 6f 6e 74 72 eeper._tcp.contr\n0030: 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 69 ol-plane.oxide.i\n0040: 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 04 nternal..!....).\n0050: d0 00 00 00 00 00 00 ....... kind = dns peer_addr = [::1]:56596 req_id = 4d374d74-d84b-4109-8441-9813fb991ec4
52132025-04-30T06:42:54.303ZDEBGtest_snapshot_basic (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 38825,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 24,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 97,\n 100,\n 109,\n 105,\n 110,\n 45,\n 107,\n 101,\n 101,\n 112,\n 101,\n 114,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:56596 req_id = 4d374d74-d84b-4109-8441-9813fb991ec4
52142025-04-30T06:42:54.303ZDEBGtest_snapshot_basic (store): query key key = _clickhouse-admin-keeper._tcp
52152025-04-30T06:42:54.303ZERROtest_snapshot_basic (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 38825, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } NXDOMAIN: no records found for name: "_clickhouse-admin-keeper._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:56596 req_id = 4d374d74-d84b-4109-8441-9813fb991ec4
52162025-04-30T06:42:54.303ZDEBGtest_snapshot_basic (BackgroundTasks): begin collection background_task = inventory_collection
52172025-04-30T06:42:54.303ZDEBGtest_snapshot_basic (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:41502
52182025-04-30T06:42:54.304ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/ignition
52192025-04-30T06:42:54.306ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
52202025-04-30T06:42:54.313ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
52212025-04-30T06:42:54.313ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
52222025-04-30T06:42:54.313ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {} }
52232025-04-30T06:42:54.314ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
52242025-04-30T06:42:54.319ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "c3a6ecd7-682c-48ec-9255-48c6caa84fae", "content-length": "835", "date": "Wed, 30 Apr 2025 06:42:54 GMT"} })
52252025-04-30T06:42:54.319ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/0
52262025-04-30T06:42:54.321ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
52272025-04-30T06:42:54.321ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {} }
52282025-04-30T06:42:54.322ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
52292025-04-30T06:42:54.322ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "f3c924de-c311-439d-8282-fe18ec721972", "content-length": "734", "date": "Wed, 30 Apr 2025 06:42:54 GMT"} })
52302025-04-30T06:42:54.326ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/0/component/sp/caboose?firmware_slot=0
52312025-04-30T06:42:54.333ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "782b03c9-7bac-444d-8bfb-47d6d696b570", "content-length": "111", "date": "Wed, 30 Apr 2025 06:42:54 GMT"} })
52322025-04-30T06:42:54.355ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/0/component/sp/caboose?firmware_slot=1
52332025-04-30T06:42:54.369ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "53af98a1-2ff5-49ba-b939-0ad70b5e1db7", "content-length": "111", "date": "Wed, 30 Apr 2025 06:42:54 GMT"} })
52342025-04-30T06:42:54.369ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/0/component/rot/caboose?firmware_slot=0
52352025-04-30T06:42:54.372ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "d68e816c-7b03-49b6-b7dd-a8204b98a8de", "content-length": "170", "date": "Wed, 30 Apr 2025 06:42:54 GMT"} })
52362025-04-30T06:42:54.372ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/0/component/rot/caboose?firmware_slot=1
52372025-04-30T06:42:54.375ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "22ae17ac-2812-4fb7-8422-24fa371d45b0", "content-length": "170", "date": "Wed, 30 Apr 2025 06:42:54 GMT"} })
52382025-04-30T06:42:54.375ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/0/component/stage0/caboose?firmware_slot=0
52392025-04-30T06:42:54.377ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "2f5bd2ba-f2c5-4454-810b-54a092c7d6a5", "content-length": "179", "date": "Wed, 30 Apr 2025 06:42:54 GMT"} })
52402025-04-30T06:42:54.378ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/0/component/stage0/caboose?firmware_slot=1
52412025-04-30T06:42:54.380ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "c187965e-ed61-4ad5-ae7d-195577eaa14e", "content-length": "179", "date": "Wed, 30 Apr 2025 06:42:54 GMT"} })
52422025-04-30T06:42:54.382ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/0/component/rot/cmpa
52432025-04-30T06:42:54.382ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "4e120c3a-f962-439c-80c5-9a8c14b62357", "content-length": "702", "date": "Wed, 30 Apr 2025 06:42:54 GMT"} })
52442025-04-30T06:42:54.382ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/0/component/rot/cfpa
52452025-04-30T06:42:54.383ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "849110c1-5560-43e7-b62e-473355af7c47", "content-length": "727", "date": "Wed, 30 Apr 2025 06:42:54 GMT"} })
52462025-04-30T06:42:54.384ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/0/component/rot/cfpa
52472025-04-30T06:42:54.387ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "d982f2e1-b4a9-41fb-93a9-d5b41825923d", "content-length": "729", "date": "Wed, 30 Apr 2025 06:42:54 GMT"} })
52482025-04-30T06:42:54.387ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/0/component/rot/cfpa
52492025-04-30T06:42:54.395ZDEBGtest_snapshot_basic: attempt to fetch latest inventory collection result = Ok("found")
52502025-04-30T06:42:54.395ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "ded7a20a-5310-43df-8abd-a962c086a27a", "content-length": "728", "date": "Wed, 30 Apr 2025 06:42:54 GMT"} })
52512025-04-30T06:42:54.396ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/1
52522025-04-30T06:42:54.397ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "5ac0ab4b-dc25-4e77-b51e-22ec5068f1a1", "content-length": "734", "date": "Wed, 30 Apr 2025 06:42:54 GMT"} })
52532025-04-30T06:42:54.397ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/1/component/sp/caboose?firmware_slot=0
52542025-04-30T06:42:54.399ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "7b7aa360-48f0-4174-8dcf-d744a3707322", "content-length": "111", "date": "Wed, 30 Apr 2025 06:42:54 GMT"} })
52552025-04-30T06:42:54.400ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/1/component/sp/caboose?firmware_slot=1
52562025-04-30T06:42:54.402ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "e1a5ef1a-974f-44db-8e1f-76267091716e", "content-length": "111", "date": "Wed, 30 Apr 2025 06:42:54 GMT"} })
52572025-04-30T06:42:54.403ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/1/component/rot/caboose?firmware_slot=0
52582025-04-30T06:42:54.405ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "4042cfb8-2a93-4195-aec7-53465511b771", "content-length": "170", "date": "Wed, 30 Apr 2025 06:42:54 GMT"} })
52592025-04-30T06:42:54.405ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/1/component/rot/caboose?firmware_slot=1
52602025-04-30T06:42:54.407ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "d89b3e5a-ab10-4172-a323-38336ca4ccae", "content-length": "170", "date": "Wed, 30 Apr 2025 06:42:54 GMT"} })
52612025-04-30T06:42:54.407ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/1/component/stage0/caboose?firmware_slot=0
52622025-04-30T06:42:54.410ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "a994d801-1a18-4593-8c5d-53a125a7ac73", "content-length": "179", "date": "Wed, 30 Apr 2025 06:42:54 GMT"} })
52632025-04-30T06:42:54.411ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/1/component/stage0/caboose?firmware_slot=1
52642025-04-30T06:42:54.415ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "9bbd9916-8348-46d2-94fc-1c59174f5e33", "content-length": "179", "date": "Wed, 30 Apr 2025 06:42:54 GMT"} })
52652025-04-30T06:42:54.415ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/1/component/rot/cmpa
52662025-04-30T06:42:54.416ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "f0aed786-272d-461b-9546-ba45f06833c2", "content-length": "702", "date": "Wed, 30 Apr 2025 06:42:54 GMT"} })
52672025-04-30T06:42:54.416ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/1/component/rot/cfpa
52682025-04-30T06:42:54.417ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "b06f2d7d-39bc-4f8b-ad34-7a9b58ed2c7d", "content-length": "727", "date": "Wed, 30 Apr 2025 06:42:54 GMT"} })
52692025-04-30T06:42:54.418ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/1/component/rot/cfpa
52702025-04-30T06:42:54.419ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "bc5dde17-3fc1-419b-9f1f-12a6034781c6", "content-length": "729", "date": "Wed, 30 Apr 2025 06:42:54 GMT"} })
52712025-04-30T06:42:54.419ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/1/component/rot/cfpa
52722025-04-30T06:42:54.420ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "d1697489-0c28-4c98-91f4-a07a92e840a9", "content-length": "728", "date": "Wed, 30 Apr 2025 06:42:54 GMT"} })
52732025-04-30T06:42:54.421ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/0
52742025-04-30T06:42:54.422ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "645a7530-b20b-4ef3-a51f-abd3b0a8a49a", "content-length": "723", "date": "Wed, 30 Apr 2025 06:42:54 GMT"} })
52752025-04-30T06:42:54.422ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/0/component/sp/caboose?firmware_slot=0
52762025-04-30T06:42:54.429ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "e156b2eb-b983-42d4-9e3d-badf37c4564a", "content-length": "109", "date": "Wed, 30 Apr 2025 06:42:54 GMT"} })
52772025-04-30T06:42:54.429ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/0/component/sp/caboose?firmware_slot=1
52782025-04-30T06:42:54.431ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "81605178-e51f-471e-89f2-eb321aa36be0", "content-length": "109", "date": "Wed, 30 Apr 2025 06:42:54 GMT"} })
52792025-04-30T06:42:54.432ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/0/component/rot/caboose?firmware_slot=0
52802025-04-30T06:42:54.446ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
52812025-04-30T06:42:54.447ZTRACtest_snapshot_basic (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
52822025-04-30T06:42:54.447ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
52832025-04-30T06:42:54.447ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {} }
52842025-04-30T06:42:54.456ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
52852025-04-30T06:42:54.456ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "277e76df-55aa-4c0a-a9bd-093fccb3b0cf", "content-length": "169", "date": "Wed, 30 Apr 2025 06:42:54 GMT"} })
52862025-04-30T06:42:54.456ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/0/component/rot/caboose?firmware_slot=1
52872025-04-30T06:42:54.462ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
52882025-04-30T06:42:54.475ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
52892025-04-30T06:42:54.475ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
52902025-04-30T06:42:54.475ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {} }
52912025-04-30T06:42:54.477ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
52922025-04-30T06:42:54.479ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "bb910ae6-cbae-4919-bcb8-730cc39fca99", "content-length": "169", "date": "Wed, 30 Apr 2025 06:42:54 GMT"} })
52932025-04-30T06:42:54.480ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/0/component/stage0/caboose?firmware_slot=0
52942025-04-30T06:42:54.482ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
52952025-04-30T06:42:54.483ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {} }
52962025-04-30T06:42:54.483ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
52972025-04-30T06:42:54.484ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "21ff6cd9-53c4-4e1a-87f8-9dfe0a15ef00", "content-length": "178", "date": "Wed, 30 Apr 2025 06:42:54 GMT"} })
52982025-04-30T06:42:54.485ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/0/component/stage0/caboose?firmware_slot=1
52992025-04-30T06:42:54.491ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "7639f5c0-c8e4-4bbb-b3ad-f6d44a8fc649", "content-length": "178", "date": "Wed, 30 Apr 2025 06:42:54 GMT"} })
53002025-04-30T06:42:54.491ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/0/component/rot/cmpa
53012025-04-30T06:42:54.494ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "e1e042c3-353b-45cc-92df-2e54dcbfc9c3", "content-length": "702", "date": "Wed, 30 Apr 2025 06:42:54 GMT"} })
53022025-04-30T06:42:54.494ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/0/component/rot/cfpa
53032025-04-30T06:42:54.496ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "d471627e-bb29-4c95-a976-efa1d85ace29", "content-length": "727", "date": "Wed, 30 Apr 2025 06:42:54 GMT"} })
53042025-04-30T06:42:54.498ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/0/component/rot/cfpa
53052025-04-30T06:42:54.500ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "3ad9e381-aa21-4ae1-a7dc-bbb7370b6483", "content-length": "729", "date": "Wed, 30 Apr 2025 06:42:54 GMT"} })
53062025-04-30T06:42:54.500ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/0/component/rot/cfpa
53072025-04-30T06:42:54.502ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "ed90a367-52ff-47a4-9778-2f6d7aed271f", "content-length": "728", "date": "Wed, 30 Apr 2025 06:42:54 GMT"} })
53082025-04-30T06:42:54.502ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/1
53092025-04-30T06:42:54.509ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "0134c226-4b6c-4039-b5bb-2052ede764db", "content-length": "723", "date": "Wed, 30 Apr 2025 06:42:54 GMT"} })
53102025-04-30T06:42:54.509ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/1/component/sp/caboose?firmware_slot=0
53112025-04-30T06:42:54.579ZDEBGtest_snapshot_basic: attempt to fetch latest inventory collection result = Ok("found")
53122025-04-30T06:42:54.597ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "699de17d-13a0-4283-88a8-9b6a9af85eb9", "content-length": "109", "date": "Wed, 30 Apr 2025 06:42:54 GMT"} })
53132025-04-30T06:42:54.597ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/1/component/sp/caboose?firmware_slot=1
53142025-04-30T06:42:54.623ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "50fd47c6-71b9-4f84-a217-2b90f6e1118b", "content-length": "109", "date": "Wed, 30 Apr 2025 06:42:54 GMT"} })
53152025-04-30T06:42:54.623ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/1/component/rot/caboose?firmware_slot=0
53162025-04-30T06:42:54.627ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "9405ed88-3f90-4f31-b110-abfd22716fb8", "content-length": "169", "date": "Wed, 30 Apr 2025 06:42:54 GMT"} })
53172025-04-30T06:42:54.628ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/1/component/rot/caboose?firmware_slot=1
53182025-04-30T06:42:54.632ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
53192025-04-30T06:42:54.632ZTRACtest_snapshot_basic (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
53202025-04-30T06:42:54.632ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
53212025-04-30T06:42:54.632ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {} }
53222025-04-30T06:42:54.632ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
53232025-04-30T06:42:54.632ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "a9410cf6-1833-48e1-9e3f-ec7cbb6a4181", "content-length": "169", "date": "Wed, 30 Apr 2025 06:42:54 GMT"} })
53242025-04-30T06:42:54.634ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/1/component/stage0/caboose?firmware_slot=0
53252025-04-30T06:42:54.634ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
53262025-04-30T06:42:54.639ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
53272025-04-30T06:42:54.639ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
53282025-04-30T06:42:54.639ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {} }
53292025-04-30T06:42:54.639ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
53302025-04-30T06:42:54.641ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "c6e24959-6348-45f5-ac90-6970bcad14bb", "content-length": "178", "date": "Wed, 30 Apr 2025 06:42:54 GMT"} })
53312025-04-30T06:42:54.641ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/1/component/stage0/caboose?firmware_slot=1
53322025-04-30T06:42:54.642ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
53332025-04-30T06:42:54.642ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {} }
53342025-04-30T06:42:54.649ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
53352025-04-30T06:42:54.651ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "32a2150d-667c-4fa6-b10f-be03331dd956", "content-length": "178", "date": "Wed, 30 Apr 2025 06:42:54 GMT"} })
53362025-04-30T06:42:54.651ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/1/component/rot/cmpa
53372025-04-30T06:42:54.652ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "4a859ef4-c1f5-4c9a-9f26-aabb37af8441", "content-length": "702", "date": "Wed, 30 Apr 2025 06:42:54 GMT"} })
53382025-04-30T06:42:54.652ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/1/component/rot/cfpa
53392025-04-30T06:42:54.654ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "dceb4cae-643b-4f4f-9831-6626f0f647a9", "content-length": "727", "date": "Wed, 30 Apr 2025 06:42:54 GMT"} })
53402025-04-30T06:42:54.655ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/1/component/rot/cfpa
53412025-04-30T06:42:54.656ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "ba4e6e78-83f3-4fce-a05c-55a8b7bf6730", "content-length": "729", "date": "Wed, 30 Apr 2025 06:42:54 GMT"} })
53422025-04-30T06:42:54.656ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/1/component/rot/cfpa
53432025-04-30T06:42:54.659ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "db4729d0-09d8-4144-8734-6314294ee743", "content-length": "728", "date": "Wed, 30 Apr 2025 06:42:54 GMT"} })
53442025-04-30T06:42:54.660ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
53452025-04-30T06:42:54.660ZTRACtest_snapshot_basic (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
53462025-04-30T06:42:54.660ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
53472025-04-30T06:42:54.660ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
53482025-04-30T06:42:54.660ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
53492025-04-30T06:42:54.665ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
53502025-04-30T06:42:54.689ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
53512025-04-30T06:42:54.689ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
53522025-04-30T06:42:54.689ZTRACtest_snapshot_basic (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
53532025-04-30T06:42:54.689ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
53542025-04-30T06:42:54.689ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
53552025-04-30T06:42:54.690ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
53562025-04-30T06:42:54.694ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
53572025-04-30T06:42:54.699ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
53582025-04-30T06:42:54.701ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
53592025-04-30T06:42:54.701ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
53602025-04-30T06:42:54.702ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
53612025-04-30T06:42:54.855ZDEBGtest_snapshot_basic (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:41923
53622025-04-30T06:42:54.855ZDEBGtest_snapshot_basic (BackgroundTasks): client request SledAgent = http://[::1]:41923 background_task = inventory_collection body = None method = GET uri = http://[::1]:41923/inventory
53632025-04-30T06:42:54.856ZINFOtest_snapshot_basic (dropshot): accepted connection local_addr = [::1]:41923 remote_addr = [::1]:56382 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
53642025-04-30T06:42:54.856ZDEBGtest_snapshot_basic (collection-task): sent timer-based collection request to the collection task collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
53652025-04-30T06:42:54.856ZDEBGtest_snapshot_basic (collection-task): collecting from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
53662025-04-30T06:42:54.858ZTRACtest_snapshot_basic (dropshot): incoming request local_addr = [::1]:41923 method = GET remote_addr = [::1]:56382 req_id = 05583d78-7f84-4617-8b59-684f51665210 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
53672025-04-30T06:42:54.860ZINFOtest_snapshot_basic (dropshot): request completed latency_us = 2011 local_addr = [::1]:41923 method = GET remote_addr = [::1]:56382 req_id = 05583d78-7f84-4617-8b59-684f51665210 response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
53682025-04-30T06:42:54.861ZTRACtest_snapshot_basic (collection-task): sent collection request to producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
53692025-04-30T06:42:54.863ZDEBGtest_snapshot_basic (collection-task): collected results from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 n_results = 1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
53702025-04-30T06:42:54.863ZTRACtest_snapshot_basic (collection-task): forwarded results to main collection loop collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
53712025-04-30T06:42:54.863ZTRACtest_snapshot_basic (collection-task): top of inner collection loop, waiting for next request collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
53722025-04-30T06:42:54.863ZDEBGtest_snapshot_basic (BackgroundTasks): client response SledAgent = http://[::1]:41923 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:41923/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "05583d78-7f84-4617-8b59-684f51665210", "content-length": "3141", "date": "Wed, 30 Apr 2025 06:42:54 GMT"} })
53732025-04-30T06:42:54.904ZDEBGtest_snapshot_basic (BackgroundTasks): begin collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
53742025-04-30T06:42:54.904ZDEBGtest_snapshot_basic (BackgroundTasks): end collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
53752025-04-30T06:42:54.904ZDEBGtest_snapshot_basic (BackgroundTasks): finished collection background_task = inventory_collection
53762025-04-30T06:42:54.907ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
53772025-04-30T06:42:54.907ZTRACtest_snapshot_basic (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
53782025-04-30T06:42:54.907ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
53792025-04-30T06:42:54.907ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
53802025-04-30T06:42:54.908ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
53812025-04-30T06:42:54.911ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
53822025-04-30T06:42:54.916ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
53832025-04-30T06:42:54.918ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
53842025-04-30T06:42:54.918ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
53852025-04-30T06:42:54.918ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
53862025-04-30T06:42:54.990ZDEBGtest_snapshot_basic: attempt to fetch latest inventory collection result = Ok("found")
53872025-04-30T06:42:55.037ZDEBGtest_snapshot_basic (results-sink): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
53882025-04-30T06:42:55.037ZDEBGtest_snapshot_basic (results-sink-cluster): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
53892025-04-30T06:42:55.037ZDEBGtest_snapshot_basic (clickhouse-client): unrolling 1 total samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 6ad06c6b-9a6e-4ed1-b3f4-5d2e9da3f8a7
53902025-04-30T06:42:55.038ZTRACtest_snapshot_basic (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 6ad06c6b-9a6e-4ed1-b3f4-5d2e9da3f8a7 n_columns = 4 n_rows = 2 sql = INSERT INTO oximeter.fields_string FORMAT Native
53912025-04-30T06:42:55.041ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
53922025-04-30T06:42:55.041ZTRACtest_snapshot_basic (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
53932025-04-30T06:42:55.041ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
53942025-04-30T06:42:55.041ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {} }
53952025-04-30T06:42:55.041ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
53962025-04-30T06:42:55.086ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
53972025-04-30T06:42:55.094ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
53982025-04-30T06:42:55.094ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
53992025-04-30T06:42:55.094ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {} }
54002025-04-30T06:42:55.094ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
54012025-04-30T06:42:55.096ZWARNtest_snapshot_basic (datastore): Non-retryable transaction failure err = DatabaseError(UniqueViolation, "duplicate key value violates unique constraint \\"inv_caboose_pkey\\"") transaction = inventory_insert_collection
54022025-04-30T06:42:55.100ZWARNtest_snapshot_basic (BackgroundTasks): inventory collection failed background_task = inventory_collection error = failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint "inv_caboose_pkey"
54032025-04-30T06:42:55.100ZDEBGtest_snapshot_basic (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 887.724162ms iteration = 13 status = {"error":"failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint \\"inv_caboose_pkey\\""}
54042025-04-30T06:42:55.100ZDEBGtest_snapshot_basic (BackgroundTasks): activating background_task = inventory_collection iteration = 14 reason = Signaled
54052025-04-30T06:42:55.100ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
54062025-04-30T06:42:55.100ZTRACtest_snapshot_basic (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
54072025-04-30T06:42:55.100ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
54082025-04-30T06:42:55.100ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
54092025-04-30T06:42:55.101ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
54102025-04-30T06:42:55.102ZDEBGtest_snapshot_basic (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 6ad06c6b-9a6e-4ed1-b3f4-5d2e9da3f8a7 n_rows = 2 table_name = fields_string
54112025-04-30T06:42:55.117ZTRACtest_snapshot_basic (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 6ad06c6b-9a6e-4ed1-b3f4-5d2e9da3f8a7 n_columns = 4 n_rows = 1 sql = INSERT INTO oximeter.measurements_i64 FORMAT Native
54122025-04-30T06:42:55.118ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
54132025-04-30T06:42:55.118ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {} }
54142025-04-30T06:42:55.118ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
54152025-04-30T06:42:55.119ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
54162025-04-30T06:42:55.123ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
54172025-04-30T06:42:55.123ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
54182025-04-30T06:42:55.123ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
54192025-04-30T06:42:55.124ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
54202025-04-30T06:42:55.125ZDEBGtest_snapshot_basic (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 6ad06c6b-9a6e-4ed1-b3f4-5d2e9da3f8a7 n_rows = 1 table_name = measurements_i64
54212025-04-30T06:42:55.125ZTRACtest_snapshot_basic (results-sink): successfully inserted samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
54222025-04-30T06:42:55.131ZDEBGtest_snapshot_basic (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = [(d1ad44e5-18dd-4193-a70e-bf9042f0e0e1, 0)]
54232025-04-30T06:42:55.131ZTRACtest_snapshot_basic (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
54242025-04-30T06:42:55.132ZTRACtest_snapshot_basic (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: a5 30 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 .0..........._mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:63911 req_id = 2b7adb9a-6379-41c2-a792-431686030820
54252025-04-30T06:42:55.132ZDEBGtest_snapshot_basic (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 42288,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:63911 req_id = 2b7adb9a-6379-41c2-a792-431686030820
54262025-04-30T06:42:55.133ZDEBGtest_snapshot_basic (store): query key key = _mgs._tcp
54272025-04-30T06:42:55.133ZDEBGtest_snapshot_basic (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
54282025-04-30T06:42:55.133ZDEBGtest_snapshot_basic (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:63911 query = LowerQuery { name: LowerName(Name("_mgs._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 41502, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = 2b7adb9a-6379-41c2-a792-431686030820
54292025-04-30T06:42:55.133ZDEBGtest_snapshot_basic (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 41502, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 4239, tv_nsec: 932749034 } })
54302025-04-30T06:42:55.133ZTRACtest_snapshot_basic (DnsResolver): lookup_service_targets: looking up SRV target
54312025-04-30T06:42:55.134ZTRACtest_snapshot_basic (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: b4 cc 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e ............-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:58983 req_id = ff5cfad1-c1ff-4513-b760-eceb36d859fe
54322025-04-30T06:42:55.134ZDEBGtest_snapshot_basic (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 46284,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:58983 req_id = ff5cfad1-c1ff-4513-b760-eceb36d859fe
54332025-04-30T06:42:55.134ZDEBGtest_snapshot_basic (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
54342025-04-30T06:42:55.134ZDEBGtest_snapshot_basic (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:58983 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = ff5cfad1-c1ff-4513-b760-eceb36d859fe
54352025-04-30T06:42:55.199ZTRACtest_snapshot_basic (DnsResolver): lookup_all_socket_v6 srv dns_name = _clickhouse-admin-keeper._tcp.control-plane.oxide.internal
54362025-04-30T06:42:55.200ZWARNtest_snapshot_basic (results-sink-cluster): failed to insert some results into metric DB collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 error = Failed to check out connection to database: No backends found for this service file = oximeter/collector/src/results_sink.rs:92
54372025-04-30T06:42:55.202ZTRACtest_snapshot_basic (dns): buffer buffer = Length: 87 (0x57) bytes\n0000: 2c 4a 01 00 00 01 00 00 00 00 00 01 18 5f 63 6c ,J..........._cl\n0010: 69 63 6b 68 6f 75 73 65 2d 61 64 6d 69 6e 2d 6b ickhouse-admin-k\n0020: 65 65 70 65 72 04 5f 74 63 70 0d 63 6f 6e 74 72 eeper._tcp.contr\n0030: 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 69 ol-plane.oxide.i\n0040: 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 04 nternal..!....).\n0050: d0 00 00 00 00 00 00 ....... kind = dns peer_addr = [::1]:49997 req_id = cdb6947b-8fd6-47f9-b075-5130aad180a3
54382025-04-30T06:42:55.203ZDEBGtest_snapshot_basic (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 11338,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 24,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 97,\n 100,\n 109,\n 105,\n 110,\n 45,\n 107,\n 101,\n 101,\n 112,\n 101,\n 114,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:49997 req_id = cdb6947b-8fd6-47f9-b075-5130aad180a3
54392025-04-30T06:42:55.203ZDEBGtest_snapshot_basic (store): query key key = _clickhouse-admin-keeper._tcp
54402025-04-30T06:42:55.203ZERROtest_snapshot_basic (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 11338, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } NXDOMAIN: no records found for name: "_clickhouse-admin-keeper._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:49997 req_id = cdb6947b-8fd6-47f9-b075-5130aad180a3
54412025-04-30T06:42:55.203ZDEBGtest_snapshot_basic (BackgroundTasks): begin collection background_task = inventory_collection
54422025-04-30T06:42:55.203ZDEBGtest_snapshot_basic (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:41502
54432025-04-30T06:42:55.203ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/ignition
54442025-04-30T06:42:55.222ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "27352404-c4d2-467d-b837-63849e70ce91", "content-length": "835", "date": "Wed, 30 Apr 2025 06:42:54 GMT"} })
54452025-04-30T06:42:55.222ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/0
54462025-04-30T06:42:55.228ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "d51257c2-356e-4617-82d5-ab6497b5f9c8", "content-length": "734", "date": "Wed, 30 Apr 2025 06:42:54 GMT"} })
54472025-04-30T06:42:55.228ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/0/component/sp/caboose?firmware_slot=0
54482025-04-30T06:42:55.234ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "ab0620a5-2f52-4a2a-a3ba-41b38a04d5ce", "content-length": "111", "date": "Wed, 30 Apr 2025 06:42:54 GMT"} })
54492025-04-30T06:42:55.234ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/0/component/sp/caboose?firmware_slot=1
54502025-04-30T06:42:55.246ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "3dc026db-41c7-418f-9e87-282cfecf6622", "content-length": "111", "date": "Wed, 30 Apr 2025 06:42:54 GMT"} })
54512025-04-30T06:42:55.246ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/0/component/rot/caboose?firmware_slot=0
54522025-04-30T06:42:55.247ZDEBGtest_snapshot_basic: attempt to fetch latest inventory collection result = Ok("found")
54532025-04-30T06:42:55.250ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "62eba57f-7e6b-48a9-ac59-0c33cdc3012f", "content-length": "170", "date": "Wed, 30 Apr 2025 06:42:54 GMT"} })
54542025-04-30T06:42:55.251ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/0/component/rot/caboose?firmware_slot=1
54552025-04-30T06:42:55.252ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "be3343ca-0da9-48b0-8e7d-ef0304d9afd0", "content-length": "170", "date": "Wed, 30 Apr 2025 06:42:54 GMT"} })
54562025-04-30T06:42:55.252ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/0/component/stage0/caboose?firmware_slot=0
54572025-04-30T06:42:55.266ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "964d83ae-9008-49ce-a3d7-bc653f98a378", "content-length": "179", "date": "Wed, 30 Apr 2025 06:42:54 GMT"} })
54582025-04-30T06:42:55.266ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/0/component/stage0/caboose?firmware_slot=1
54592025-04-30T06:42:55.272ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "58bfca31-be0a-4a10-bc35-6fedf53ea359", "content-length": "179", "date": "Wed, 30 Apr 2025 06:42:54 GMT"} })
54602025-04-30T06:42:55.272ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/0/component/rot/cmpa
54612025-04-30T06:42:55.274ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "c30f38e9-ef86-40fc-a15a-ebb3480c6496", "content-length": "702", "date": "Wed, 30 Apr 2025 06:42:54 GMT"} })
54622025-04-30T06:42:55.274ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/0/component/rot/cfpa
54632025-04-30T06:42:55.277ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "8e48d6eb-c14e-4349-b37b-4ee236d30d85", "content-length": "727", "date": "Wed, 30 Apr 2025 06:42:54 GMT"} })
54642025-04-30T06:42:55.278ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/0/component/rot/cfpa
54652025-04-30T06:42:55.280ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "2c2a002a-0f4d-4aca-89a2-cf9af54730e0", "content-length": "729", "date": "Wed, 30 Apr 2025 06:42:54 GMT"} })
54662025-04-30T06:42:55.280ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/0/component/rot/cfpa
54672025-04-30T06:42:55.283ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "3192f7fd-27c8-4dc3-bede-e2044ff58f47", "content-length": "728", "date": "Wed, 30 Apr 2025 06:42:54 GMT"} })
54682025-04-30T06:42:55.283ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/1
54692025-04-30T06:42:55.288ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "657c92c4-517e-4064-beef-51673fb70a57", "content-length": "734", "date": "Wed, 30 Apr 2025 06:42:54 GMT"} })
54702025-04-30T06:42:55.288ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/1/component/sp/caboose?firmware_slot=0
54712025-04-30T06:42:55.290ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "f2d04265-02e3-4ee5-b5cc-15e89a0dab78", "content-length": "111", "date": "Wed, 30 Apr 2025 06:42:54 GMT"} })
54722025-04-30T06:42:55.290ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/1/component/sp/caboose?firmware_slot=1
54732025-04-30T06:42:55.292ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "5a3504c2-b301-4abf-a8a4-42e622cb3500", "content-length": "111", "date": "Wed, 30 Apr 2025 06:42:54 GMT"} })
54742025-04-30T06:42:55.292ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/1/component/rot/caboose?firmware_slot=0
54752025-04-30T06:42:55.293ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "a21080f7-9667-459c-86c8-a6b9eef0cc3f", "content-length": "170", "date": "Wed, 30 Apr 2025 06:42:54 GMT"} })
54762025-04-30T06:42:55.293ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/1/component/rot/caboose?firmware_slot=1
54772025-04-30T06:42:55.311ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
54782025-04-30T06:42:55.311ZTRACtest_snapshot_basic (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
54792025-04-30T06:42:55.311ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
54802025-04-30T06:42:55.311ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {} }
54812025-04-30T06:42:55.312ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
54822025-04-30T06:42:55.330ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
54832025-04-30T06:42:55.336ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
54842025-04-30T06:42:55.336ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
54852025-04-30T06:42:55.336ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {} }
54862025-04-30T06:42:55.337ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
54872025-04-30T06:42:55.339ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
54882025-04-30T06:42:55.339ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {} }
54892025-04-30T06:42:55.340ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
54902025-04-30T06:42:55.343ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "905debc7-0ef7-4c4a-aa93-21e953dccc09", "content-length": "170", "date": "Wed, 30 Apr 2025 06:42:55 GMT"} })
54912025-04-30T06:42:55.343ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/1/component/stage0/caboose?firmware_slot=0
54922025-04-30T06:42:55.346ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "bc44e11c-03ab-4d39-b643-46ec03339cf2", "content-length": "179", "date": "Wed, 30 Apr 2025 06:42:55 GMT"} })
54932025-04-30T06:42:55.346ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/1/component/stage0/caboose?firmware_slot=1
54942025-04-30T06:42:55.348ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "24b12271-d94e-410d-8c74-3463c8db8a28", "content-length": "179", "date": "Wed, 30 Apr 2025 06:42:55 GMT"} })
54952025-04-30T06:42:55.348ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/1/component/rot/cmpa
54962025-04-30T06:42:55.350ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "d103bc5e-cf6c-4014-9ec2-e7b8468dd67a", "content-length": "702", "date": "Wed, 30 Apr 2025 06:42:55 GMT"} })
54972025-04-30T06:42:55.350ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/1/component/rot/cfpa
54982025-04-30T06:42:55.352ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "e62de1f8-d402-4e56-9e6d-b97086214089", "content-length": "727", "date": "Wed, 30 Apr 2025 06:42:55 GMT"} })
54992025-04-30T06:42:55.352ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/1/component/rot/cfpa
55002025-04-30T06:42:55.355ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "bc353d7a-9dc6-46e0-b032-7c66fe098835", "content-length": "729", "date": "Wed, 30 Apr 2025 06:42:55 GMT"} })
55012025-04-30T06:42:55.355ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/1/component/rot/cfpa
55022025-04-30T06:42:55.356ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "b586f696-ea40-4456-9926-1e188465dd58", "content-length": "728", "date": "Wed, 30 Apr 2025 06:42:55 GMT"} })
55032025-04-30T06:42:55.356ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/0
55042025-04-30T06:42:55.359ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "a7c50dc2-0769-4de9-b38d-f93bad599404", "content-length": "723", "date": "Wed, 30 Apr 2025 06:42:55 GMT"} })
55052025-04-30T06:42:55.359ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/0/component/sp/caboose?firmware_slot=0
55062025-04-30T06:42:55.361ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "35b02e3e-941f-49a9-98a3-7a2991b6b6a2", "content-length": "109", "date": "Wed, 30 Apr 2025 06:42:55 GMT"} })
55072025-04-30T06:42:55.361ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/0/component/sp/caboose?firmware_slot=1
55082025-04-30T06:42:55.366ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "0d113d05-48d2-47c9-9e07-902ccb944c47", "content-length": "109", "date": "Wed, 30 Apr 2025 06:42:55 GMT"} })
55092025-04-30T06:42:55.366ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/0/component/rot/caboose?firmware_slot=0
55102025-04-30T06:42:55.373ZDEBGtest_snapshot_basic: attempt to fetch latest inventory collection result = Ok("found")
55112025-04-30T06:42:55.375ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "acd6a4c8-b1e4-4f2d-bc5e-09ded2288c7b", "content-length": "169", "date": "Wed, 30 Apr 2025 06:42:55 GMT"} })
55122025-04-30T06:42:55.375ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/0/component/rot/caboose?firmware_slot=1
55132025-04-30T06:42:55.377ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "7d42b7d9-d724-4454-ba99-d3af75a73a09", "content-length": "169", "date": "Wed, 30 Apr 2025 06:42:55 GMT"} })
55142025-04-30T06:42:55.382ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/0/component/stage0/caboose?firmware_slot=0
55152025-04-30T06:42:55.387ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "de80e137-9f7e-48bf-90fa-10a226f3c9fe", "content-length": "178", "date": "Wed, 30 Apr 2025 06:42:55 GMT"} })
55162025-04-30T06:42:55.387ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/0/component/stage0/caboose?firmware_slot=1
55172025-04-30T06:42:55.396ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "b9397f96-8acf-4c1b-abeb-e92b69fbdf14", "content-length": "178", "date": "Wed, 30 Apr 2025 06:42:55 GMT"} })
55182025-04-30T06:42:55.396ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/0/component/rot/cmpa
55192025-04-30T06:42:55.397ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "89c77a2b-471b-490d-af1e-47c07255c674", "content-length": "702", "date": "Wed, 30 Apr 2025 06:42:55 GMT"} })
55202025-04-30T06:42:55.397ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/0/component/rot/cfpa
55212025-04-30T06:42:55.399ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "7543dbdb-be05-4b43-88e9-1aa9e1c6c9e2", "content-length": "727", "date": "Wed, 30 Apr 2025 06:42:55 GMT"} })
55222025-04-30T06:42:55.399ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/0/component/rot/cfpa
55232025-04-30T06:42:55.400ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "7531b3df-121a-44cb-bba2-3cdfe00f769e", "content-length": "729", "date": "Wed, 30 Apr 2025 06:42:55 GMT"} })
55242025-04-30T06:42:55.400ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/0/component/rot/cfpa
55252025-04-30T06:42:55.401ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "db0c6fc8-6f31-4c93-82ed-3673f498b9e0", "content-length": "728", "date": "Wed, 30 Apr 2025 06:42:55 GMT"} })
55262025-04-30T06:42:55.402ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/1
55272025-04-30T06:42:55.411ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "afab9652-85f7-4d79-9c29-e55865bb9030", "content-length": "723", "date": "Wed, 30 Apr 2025 06:42:55 GMT"} })
55282025-04-30T06:42:55.414ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/1/component/sp/caboose?firmware_slot=0
55292025-04-30T06:42:55.420ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "b40bb3a1-e8bf-4443-bdc1-3b6d3e7c2226", "content-length": "109", "date": "Wed, 30 Apr 2025 06:42:55 GMT"} })
55302025-04-30T06:42:55.439ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/1/component/sp/caboose?firmware_slot=1
55312025-04-30T06:42:55.440ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
55322025-04-30T06:42:55.440ZTRACtest_snapshot_basic (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
55332025-04-30T06:42:55.440ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
55342025-04-30T06:42:55.440ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {} }
55352025-04-30T06:42:55.441ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
55362025-04-30T06:42:55.443ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "746c2b22-ac4c-44a8-b44e-28fe00c678ee", "content-length": "109", "date": "Wed, 30 Apr 2025 06:42:55 GMT"} })
55372025-04-30T06:42:55.443ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/1/component/rot/caboose?firmware_slot=0
55382025-04-30T06:42:55.444ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
55392025-04-30T06:42:55.449ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
55402025-04-30T06:42:55.449ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
55412025-04-30T06:42:55.449ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {} }
55422025-04-30T06:42:55.449ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
55432025-04-30T06:42:55.452ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
55442025-04-30T06:42:55.452ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {} }
55452025-04-30T06:42:55.452ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
55462025-04-30T06:42:55.454ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "c65fe80c-ec02-493f-b3c3-f9f8aa7ffe9d", "content-length": "169", "date": "Wed, 30 Apr 2025 06:42:55 GMT"} })
55472025-04-30T06:42:55.455ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/1/component/rot/caboose?firmware_slot=1
55482025-04-30T06:42:55.461ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "c6627919-c212-4dc2-9787-9beec1dcfd86", "content-length": "169", "date": "Wed, 30 Apr 2025 06:42:55 GMT"} })
55492025-04-30T06:42:55.461ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/1/component/stage0/caboose?firmware_slot=0
55502025-04-30T06:42:55.467ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "15ec7221-1c2e-4503-856a-549cb6065b21", "content-length": "178", "date": "Wed, 30 Apr 2025 06:42:55 GMT"} })
55512025-04-30T06:42:55.467ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/1/component/stage0/caboose?firmware_slot=1
55522025-04-30T06:42:55.471ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "e179b112-c66e-46c9-b83b-9c0b09c638d9", "content-length": "178", "date": "Wed, 30 Apr 2025 06:42:55 GMT"} })
55532025-04-30T06:42:55.471ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/1/component/rot/cmpa
55542025-04-30T06:42:55.477ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "0ae239de-5f4c-4e1c-8bb7-7a60d15d29b1", "content-length": "702", "date": "Wed, 30 Apr 2025 06:42:55 GMT"} })
55552025-04-30T06:42:55.477ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/1/component/rot/cfpa
55562025-04-30T06:42:55.478ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "32e6cc26-1929-4545-b738-fe333fee94e1", "content-length": "727", "date": "Wed, 30 Apr 2025 06:42:55 GMT"} })
55572025-04-30T06:42:55.478ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/1/component/rot/cfpa
55582025-04-30T06:42:55.481ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "a837365a-ced9-4829-9fc0-59c1fe27b7e4", "content-length": "729", "date": "Wed, 30 Apr 2025 06:42:55 GMT"} })
55592025-04-30T06:42:55.481ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/1/component/rot/cfpa
55602025-04-30T06:42:55.482ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "f53880a3-dc5b-4725-b685-1e013359b4f0", "content-length": "728", "date": "Wed, 30 Apr 2025 06:42:55 GMT"} })
55612025-04-30T06:42:55.482ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
55622025-04-30T06:42:55.482ZTRACtest_snapshot_basic (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
55632025-04-30T06:42:55.482ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
55642025-04-30T06:42:55.482ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
55652025-04-30T06:42:55.482ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
55662025-04-30T06:42:55.490ZDEBGtest_snapshot_basic: attempt to fetch latest inventory collection result = Ok("found")
55672025-04-30T06:42:55.493ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
55682025-04-30T06:42:55.497ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
55692025-04-30T06:42:55.497ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
55702025-04-30T06:42:55.497ZTRACtest_snapshot_basic (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
55712025-04-30T06:42:55.497ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
55722025-04-30T06:42:55.497ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
55732025-04-30T06:42:55.498ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
55742025-04-30T06:42:55.503ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
55752025-04-30T06:42:55.523ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
55762025-04-30T06:42:55.524ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
55772025-04-30T06:42:55.524ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
55782025-04-30T06:42:55.525ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
55792025-04-30T06:42:55.657ZDEBGtest_snapshot_basic (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:41923
55802025-04-30T06:42:55.657ZDEBGtest_snapshot_basic (BackgroundTasks): client request SledAgent = http://[::1]:41923 background_task = inventory_collection body = None method = GET uri = http://[::1]:41923/inventory
55812025-04-30T06:42:55.658ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
55822025-04-30T06:42:55.658ZTRACtest_snapshot_basic (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
55832025-04-30T06:42:55.658ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
55842025-04-30T06:42:55.658ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {} }
55852025-04-30T06:42:55.659ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
55862025-04-30T06:42:55.660ZINFOtest_snapshot_basic (dropshot): accepted connection local_addr = [::1]:41923 remote_addr = [::1]:45584 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
55872025-04-30T06:42:55.661ZTRACtest_snapshot_basic (dropshot): incoming request local_addr = [::1]:41923 method = GET remote_addr = [::1]:45584 req_id = fc4cf82e-ece9-428a-8158-6936adbe1ffb sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
55882025-04-30T06:42:55.672ZINFOtest_snapshot_basic (dropshot): request completed latency_us = 11491 local_addr = [::1]:41923 method = GET remote_addr = [::1]:45584 req_id = fc4cf82e-ece9-428a-8158-6936adbe1ffb response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
55892025-04-30T06:42:55.675ZDEBGtest_snapshot_basic (BackgroundTasks): client response SledAgent = http://[::1]:41923 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:41923/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "fc4cf82e-ece9-428a-8158-6936adbe1ffb", "content-length": "3141", "date": "Wed, 30 Apr 2025 06:42:55 GMT"} })
55902025-04-30T06:42:55.679ZDEBGtest_snapshot_basic (BackgroundTasks): begin collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
55912025-04-30T06:42:55.679ZDEBGtest_snapshot_basic (BackgroundTasks): end collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
55922025-04-30T06:42:55.679ZDEBGtest_snapshot_basic (BackgroundTasks): finished collection background_task = inventory_collection
55932025-04-30T06:42:55.682ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
55942025-04-30T06:42:55.682ZTRACtest_snapshot_basic (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
55952025-04-30T06:42:55.682ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
55962025-04-30T06:42:55.682ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
55972025-04-30T06:42:55.682ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
55982025-04-30T06:42:55.683ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
55992025-04-30T06:42:55.689ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
56002025-04-30T06:42:55.690ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
56012025-04-30T06:42:55.690ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {} }
56022025-04-30T06:42:55.690ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
56032025-04-30T06:42:55.694ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
56042025-04-30T06:42:55.699ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
56052025-04-30T06:42:55.699ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
56062025-04-30T06:42:55.699ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
56072025-04-30T06:42:55.700ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
56082025-04-30T06:42:55.707ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
56092025-04-30T06:42:55.708ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {} }
56102025-04-30T06:42:55.709ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
56112025-04-30T06:42:55.729ZINFOtest_snapshot_basic (oximeter-agent): refreshing list of producers from Nexus collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 file = oximeter/collector/src/agent.rs:562
56122025-04-30T06:42:55.729ZDEBGtest_snapshot_basic (oximeter): client request body = None method = GET uri = http://127.0.0.1:61468/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
56132025-04-30T06:42:55.730ZTRACtest_snapshot_basic (dropshot_internal): incoming request local_addr = 127.0.0.1:61468 method = GET remote_addr = 127.0.0.1:64332 req_id = fcfd477d-1d68-4602-9244-68ce4d8e0592 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
56142025-04-30T06:42:55.732ZTRACtest_snapshot_basic (dropshot_internal): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:61468 method = GET remote_addr = 127.0.0.1:64332 req_id = fcfd477d-1d68-4602-9244-68ce4d8e0592 resource = Database uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
56152025-04-30T06:42:55.732ZDEBGtest_snapshot_basic (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:61468 method = GET remote_addr = 127.0.0.1:64332 req_id = fcfd477d-1d68-4602-9244-68ce4d8e0592 roles = RoleSet { roles: {} } uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
56162025-04-30T06:42:55.733ZDEBGtest_snapshot_basic (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:61468 method = GET remote_addr = 127.0.0.1:64332 req_id = fcfd477d-1d68-4602-9244-68ce4d8e0592 resource = Database result = Ok(()) uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
56172025-04-30T06:42:55.738ZINFOtest_snapshot_basic (dropshot_internal): request completed latency_us = 8040 local_addr = 127.0.0.1:61468 method = GET remote_addr = 127.0.0.1:64332 req_id = fcfd477d-1d68-4602-9244-68ce4d8e0592 response_code = 200 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
56182025-04-30T06:42:55.741ZDEBGtest_snapshot_basic (oximeter): client response result = Ok(Response { url: "http://127.0.0.1:61468/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending", status: 200, headers: {"content-type": "application/json", "x-request-id": "fcfd477d-1d68-4602-9244-68ce4d8e0592", "content-length": "405", "date": "Wed, 30 Apr 2025 06:42:55 GMT"} })
56192025-04-30T06:42:55.741ZDEBGtest_snapshot_basic (oximeter): client request body = None method = GET uri = http://127.0.0.1:61468/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
56202025-04-30T06:42:55.741ZTRACtest_snapshot_basic (dropshot_internal): incoming request local_addr = 127.0.0.1:61468 method = GET remote_addr = 127.0.0.1:64332 req_id = dfaac301-9acd-448f-8d5b-d971fe74e3b3 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
56212025-04-30T06:42:55.742ZTRACtest_snapshot_basic (dropshot_internal): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:61468 method = GET remote_addr = 127.0.0.1:64332 req_id = dfaac301-9acd-448f-8d5b-d971fe74e3b3 resource = Database uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
56222025-04-30T06:42:55.742ZDEBGtest_snapshot_basic (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:61468 method = GET remote_addr = 127.0.0.1:64332 req_id = dfaac301-9acd-448f-8d5b-d971fe74e3b3 roles = RoleSet { roles: {} } uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
56232025-04-30T06:42:55.743ZDEBGtest_snapshot_basic (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:61468 method = GET remote_addr = 127.0.0.1:64332 req_id = dfaac301-9acd-448f-8d5b-d971fe74e3b3 resource = Database result = Ok(()) uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
56242025-04-30T06:42:55.746ZINFOtest_snapshot_basic (dropshot_internal): request completed latency_us = 4600 local_addr = 127.0.0.1:61468 method = GET remote_addr = 127.0.0.1:64332 req_id = dfaac301-9acd-448f-8d5b-d971fe74e3b3 response_code = 200 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
56252025-04-30T06:42:55.747ZDEBGtest_snapshot_basic (oximeter): client response result = Ok(Response { url: "http://127.0.0.1:61468/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D", status: 200, headers: {"content-type": "application/json", "x-request-id": "dfaac301-9acd-448f-8d5b-d971fe74e3b3", "content-length": "29", "date": "Wed, 30 Apr 2025 06:42:55 GMT"} })
56262025-04-30T06:42:55.747ZTRACtest_snapshot_basic (oximeter-agent): ignoring request to update existing metric producer, since the endpoint information is the same as the existing collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = 354a52d4-3078-4706-81ce-12c32f6bbeaa
56272025-04-30T06:42:55.747ZTRACtest_snapshot_basic (oximeter-agent): ignoring request to update existing metric producer, since the endpoint information is the same as the existing collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
56282025-04-30T06:42:55.747ZINFOtest_snapshot_basic (oximeter-agent): refreshed list of producers from Nexus collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 file = oximeter/collector/src/agent.rs:637 n_current_tasks = 2 n_pruned_tasks = 0
56292025-04-30T06:42:55.750ZWARNtest_snapshot_basic (datastore): Non-retryable transaction failure err = DatabaseError(UniqueViolation, "duplicate key value violates unique constraint \\"inv_caboose_pkey\\"") transaction = inventory_insert_collection
56302025-04-30T06:42:55.750ZWARNtest_snapshot_basic (BackgroundTasks): inventory collection failed background_task = inventory_collection error = failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint "inv_caboose_pkey"
56312025-04-30T06:42:55.750ZDEBGtest_snapshot_basic (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 650.270472ms iteration = 14 status = {"error":"failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint \\"inv_caboose_pkey\\""}
56322025-04-30T06:42:55.751ZDEBGtest_snapshot_basic (BackgroundTasks): activating background_task = inventory_collection iteration = 15 reason = Signaled
56332025-04-30T06:42:55.751ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
56342025-04-30T06:42:55.751ZTRACtest_snapshot_basic (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
56352025-04-30T06:42:55.751ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
56362025-04-30T06:42:55.751ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
56372025-04-30T06:42:55.751ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
56382025-04-30T06:42:55.764ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
56392025-04-30T06:42:55.771ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
56402025-04-30T06:42:55.771ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
56412025-04-30T06:42:55.771ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
56422025-04-30T06:42:55.772ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
56432025-04-30T06:42:55.776ZDEBGtest_snapshot_basic (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = [(d1ad44e5-18dd-4193-a70e-bf9042f0e0e1, 0)]
56442025-04-30T06:42:55.776ZTRACtest_snapshot_basic (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
56452025-04-30T06:42:55.778ZTRACtest_snapshot_basic (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: 1c a5 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 ............._mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:59511 req_id = b3aa1c57-bc02-4199-9101-04c6cc8a9dc5
56462025-04-30T06:42:55.778ZDEBGtest_snapshot_basic (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 7333,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:59511 req_id = b3aa1c57-bc02-4199-9101-04c6cc8a9dc5
56472025-04-30T06:42:55.778ZDEBGtest_snapshot_basic (store): query key key = _mgs._tcp
56482025-04-30T06:42:55.778ZDEBGtest_snapshot_basic (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
56492025-04-30T06:42:55.778ZDEBGtest_snapshot_basic (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:59511 query = LowerQuery { name: LowerName(Name("_mgs._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 41502, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = b3aa1c57-bc02-4199-9101-04c6cc8a9dc5
56502025-04-30T06:42:55.781ZDEBGtest_snapshot_basic (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 41502, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 4240, tv_nsec: 580042895 } })
56512025-04-30T06:42:55.781ZTRACtest_snapshot_basic (DnsResolver): lookup_service_targets: looking up SRV target
56522025-04-30T06:42:55.781ZTRACtest_snapshot_basic (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: b7 26 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e .&..........-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:56543 req_id = 9af4b1eb-ab41-4379-bc8f-ff44aa879e99
56532025-04-30T06:42:55.781ZDEBGtest_snapshot_basic (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 46886,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:56543 req_id = 9af4b1eb-ab41-4379-bc8f-ff44aa879e99
56542025-04-30T06:42:55.782ZDEBGtest_snapshot_basic (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
56552025-04-30T06:42:55.782ZDEBGtest_snapshot_basic (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:56543 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = 9af4b1eb-ab41-4379-bc8f-ff44aa879e99
56562025-04-30T06:42:55.782ZDEBGtest_snapshot_basic: attempt to fetch latest inventory collection result = Ok("found")
56572025-04-30T06:42:55.865ZTRACtest_snapshot_basic (DnsResolver): lookup_all_socket_v6 srv dns_name = _clickhouse-admin-keeper._tcp.control-plane.oxide.internal
56582025-04-30T06:42:55.866ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
56592025-04-30T06:42:55.866ZTRACtest_snapshot_basic (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
56602025-04-30T06:42:55.866ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
56612025-04-30T06:42:55.866ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {} }
56622025-04-30T06:42:55.867ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
56632025-04-30T06:42:55.867ZDEBGtest_snapshot_basic (collection-task): sent timer-based collection request to the collection task collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
56642025-04-30T06:42:55.867ZDEBGtest_snapshot_basic (collection-task): collecting from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
56652025-04-30T06:42:55.868ZTRACtest_snapshot_basic (dns): buffer buffer = Length: 87 (0x57) bytes\n0000: 24 ae 01 00 00 01 00 00 00 00 00 01 18 5f 63 6c $............_cl\n0010: 69 63 6b 68 6f 75 73 65 2d 61 64 6d 69 6e 2d 6b ickhouse-admin-k\n0020: 65 65 70 65 72 04 5f 74 63 70 0d 63 6f 6e 74 72 eeper._tcp.contr\n0030: 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 69 ol-plane.oxide.i\n0040: 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 04 nternal..!....).\n0050: d0 00 00 00 00 00 00 ....... kind = dns peer_addr = [::1]:55816 req_id = a6e60f4e-c718-424f-a47c-0167c23bd120
56662025-04-30T06:42:55.868ZDEBGtest_snapshot_basic (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 9390,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 24,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 97,\n 100,\n 109,\n 105,\n 110,\n 45,\n 107,\n 101,\n 101,\n 112,\n 101,\n 114,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:55816 req_id = a6e60f4e-c718-424f-a47c-0167c23bd120
56672025-04-30T06:42:55.869ZDEBGtest_snapshot_basic (store): query key key = _clickhouse-admin-keeper._tcp
56682025-04-30T06:42:55.869ZERROtest_snapshot_basic (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 9390, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } NXDOMAIN: no records found for name: "_clickhouse-admin-keeper._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:55816 req_id = a6e60f4e-c718-424f-a47c-0167c23bd120
56692025-04-30T06:42:55.870ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
56702025-04-30T06:42:55.877ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
56712025-04-30T06:42:55.877ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
56722025-04-30T06:42:55.877ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {} }
56732025-04-30T06:42:55.878ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
56742025-04-30T06:42:55.878ZDEBGtest_snapshot_basic (BackgroundTasks): begin collection background_task = inventory_collection
56752025-04-30T06:42:55.878ZDEBGtest_snapshot_basic (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:41502
56762025-04-30T06:42:55.878ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/ignition
56772025-04-30T06:42:55.879ZTRACtest_snapshot_basic (collection-task): sent collection request to producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
56782025-04-30T06:42:55.880ZDEBGtest_snapshot_basic (collection-task): collected results from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 n_results = 1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
56792025-04-30T06:42:55.880ZTRACtest_snapshot_basic (collection-task): forwarded results to main collection loop collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
56802025-04-30T06:42:55.880ZTRACtest_snapshot_basic (collection-task): top of inner collection loop, waiting for next request collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
56812025-04-30T06:42:55.882ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
56822025-04-30T06:42:55.882ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {} }
56832025-04-30T06:42:55.885ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
56842025-04-30T06:42:55.889ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "883b4117-f642-4bf5-8e77-e0b803ec0db9", "content-length": "835", "date": "Wed, 30 Apr 2025 06:42:55 GMT"} })
56852025-04-30T06:42:55.889ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/0
56862025-04-30T06:42:55.902ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "63e36e39-e93a-4b71-973d-1a076b71edae", "content-length": "734", "date": "Wed, 30 Apr 2025 06:42:55 GMT"} })
56872025-04-30T06:42:55.903ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/0/component/sp/caboose?firmware_slot=0
56882025-04-30T06:42:55.907ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "cac648a8-9bcf-40a5-b65d-6fbcfcb084fa", "content-length": "111", "date": "Wed, 30 Apr 2025 06:42:55 GMT"} })
56892025-04-30T06:42:55.908ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/0/component/sp/caboose?firmware_slot=1
56902025-04-30T06:42:55.912ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "171863d0-7a80-47ee-9b68-5f701394485e", "content-length": "111", "date": "Wed, 30 Apr 2025 06:42:55 GMT"} })
56912025-04-30T06:42:55.912ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/0/component/rot/caboose?firmware_slot=0
56922025-04-30T06:42:55.916ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "3bfb2ded-16a2-4f61-86e2-d80b232347d2", "content-length": "170", "date": "Wed, 30 Apr 2025 06:42:55 GMT"} })
56932025-04-30T06:42:55.916ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/0/component/rot/caboose?firmware_slot=1
56942025-04-30T06:42:55.921ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "461edec9-7748-4f93-a702-e50d1b5cc4c6", "content-length": "170", "date": "Wed, 30 Apr 2025 06:42:55 GMT"} })
56952025-04-30T06:42:55.921ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/0/component/stage0/caboose?firmware_slot=0
56962025-04-30T06:42:55.922ZDEBGtest_snapshot_basic: attempt to fetch latest inventory collection result = Ok("found")
56972025-04-30T06:42:55.925ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "09e34dab-1047-4ba4-b712-ece7c6eb6c36", "content-length": "179", "date": "Wed, 30 Apr 2025 06:42:55 GMT"} })
56982025-04-30T06:42:55.925ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/0/component/stage0/caboose?firmware_slot=1
56992025-04-30T06:42:55.927ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "4cad7c85-3d5a-44bd-b23c-71b8dc0fdd23", "content-length": "179", "date": "Wed, 30 Apr 2025 06:42:55 GMT"} })
57002025-04-30T06:42:55.927ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/0/component/rot/cmpa
57012025-04-30T06:42:55.928ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "fbedeb51-12a2-41e2-85cd-07d5def7b799", "content-length": "702", "date": "Wed, 30 Apr 2025 06:42:55 GMT"} })
57022025-04-30T06:42:55.928ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/0/component/rot/cfpa
57032025-04-30T06:42:55.929ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "7bf6be21-e61b-4a62-81f1-ffe624eb9f05", "content-length": "727", "date": "Wed, 30 Apr 2025 06:42:55 GMT"} })
57042025-04-30T06:42:55.929ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/0/component/rot/cfpa
57052025-04-30T06:42:55.931ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "e23b7524-597b-4352-b8b5-a544a5be9eed", "content-length": "729", "date": "Wed, 30 Apr 2025 06:42:55 GMT"} })
57062025-04-30T06:42:55.931ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/0/component/rot/cfpa
57072025-04-30T06:42:55.933ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "ea3d83e2-c09d-47c7-9beb-2b1a14599290", "content-length": "728", "date": "Wed, 30 Apr 2025 06:42:55 GMT"} })
57082025-04-30T06:42:55.933ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/1
57092025-04-30T06:42:55.935ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "7b2562a7-fcbf-453e-b5e1-5f27606ae1cc", "content-length": "734", "date": "Wed, 30 Apr 2025 06:42:55 GMT"} })
57102025-04-30T06:42:55.935ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/1/component/sp/caboose?firmware_slot=0
57112025-04-30T06:42:55.937ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "1ec954ca-6f85-4778-8510-4b7dd961da58", "content-length": "111", "date": "Wed, 30 Apr 2025 06:42:55 GMT"} })
57122025-04-30T06:42:55.937ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/1/component/sp/caboose?firmware_slot=1
57132025-04-30T06:42:55.939ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "2172f531-29c8-43b6-a4f9-e7f142e3e054", "content-length": "111", "date": "Wed, 30 Apr 2025 06:42:55 GMT"} })
57142025-04-30T06:42:55.939ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/1/component/rot/caboose?firmware_slot=0
57152025-04-30T06:42:55.941ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "9575e0d1-7b0d-4525-9f85-7648d483e85b", "content-length": "170", "date": "Wed, 30 Apr 2025 06:42:55 GMT"} })
57162025-04-30T06:42:55.941ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/1/component/rot/caboose?firmware_slot=1
57172025-04-30T06:42:55.944ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "fa2b42a3-e51c-45b8-b29a-ecb637a85dd7", "content-length": "170", "date": "Wed, 30 Apr 2025 06:42:55 GMT"} })
57182025-04-30T06:42:55.944ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/1/component/stage0/caboose?firmware_slot=0
57192025-04-30T06:42:55.947ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "7428a80a-437d-403c-90ea-8a04c865043c", "content-length": "179", "date": "Wed, 30 Apr 2025 06:42:55 GMT"} })
57202025-04-30T06:42:55.947ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/1/component/stage0/caboose?firmware_slot=1
57212025-04-30T06:42:55.949ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "05cff118-c86f-4bc0-84cf-39b84a71dac2", "content-length": "179", "date": "Wed, 30 Apr 2025 06:42:55 GMT"} })
57222025-04-30T06:42:55.949ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/1/component/rot/cmpa
57232025-04-30T06:42:55.950ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "60ada20b-4d38-4e16-a9ec-6ba423561483", "content-length": "702", "date": "Wed, 30 Apr 2025 06:42:55 GMT"} })
57242025-04-30T06:42:55.950ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/1/component/rot/cfpa
57252025-04-30T06:42:55.951ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "2aadddf4-bae3-484e-8107-0a0e5a20b652", "content-length": "727", "date": "Wed, 30 Apr 2025 06:42:55 GMT"} })
57262025-04-30T06:42:55.951ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/1/component/rot/cfpa
57272025-04-30T06:42:55.953ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "f5912f71-7211-4fd5-8d3d-c3eb9dc2efc7", "content-length": "729", "date": "Wed, 30 Apr 2025 06:42:55 GMT"} })
57282025-04-30T06:42:55.954ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/1/component/rot/cfpa
57292025-04-30T06:42:55.954ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "8f05ed61-b7a5-4ee7-bb1a-2986b694278c", "content-length": "728", "date": "Wed, 30 Apr 2025 06:42:55 GMT"} })
57302025-04-30T06:42:55.955ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/0
57312025-04-30T06:42:55.957ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "44eb85e6-1321-42bd-bd6f-48c8bfabc50e", "content-length": "723", "date": "Wed, 30 Apr 2025 06:42:55 GMT"} })
57322025-04-30T06:42:55.957ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/0/component/sp/caboose?firmware_slot=0
57332025-04-30T06:42:55.959ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "a8cbfaa4-3cbe-44f4-8596-f7597670c7d6", "content-length": "109", "date": "Wed, 30 Apr 2025 06:42:55 GMT"} })
57342025-04-30T06:42:55.959ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/0/component/sp/caboose?firmware_slot=1
57352025-04-30T06:42:55.961ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "529807df-4f47-4544-969a-5dbcc969bda9", "content-length": "109", "date": "Wed, 30 Apr 2025 06:42:55 GMT"} })
57362025-04-30T06:42:55.962ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/0/component/rot/caboose?firmware_slot=0
57372025-04-30T06:42:55.965ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "1587af8c-3f33-443e-9139-7bd73bf5ccf7", "content-length": "169", "date": "Wed, 30 Apr 2025 06:42:55 GMT"} })
57382025-04-30T06:42:55.965ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/0/component/rot/caboose?firmware_slot=1
57392025-04-30T06:42:55.967ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "7f05cb27-8ce2-430e-9150-631d21c5c087", "content-length": "169", "date": "Wed, 30 Apr 2025 06:42:55 GMT"} })
57402025-04-30T06:42:55.967ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/0/component/stage0/caboose?firmware_slot=0
57412025-04-30T06:42:55.970ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "e84f1d6b-c172-42f7-bc6e-bb75c1a477fa", "content-length": "178", "date": "Wed, 30 Apr 2025 06:42:55 GMT"} })
57422025-04-30T06:42:55.970ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/0/component/stage0/caboose?firmware_slot=1
57432025-04-30T06:42:55.975ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "85942188-6cec-477a-8902-8ea7d78417ba", "content-length": "178", "date": "Wed, 30 Apr 2025 06:42:55 GMT"} })
57442025-04-30T06:42:55.975ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/0/component/rot/cmpa
57452025-04-30T06:42:55.975ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
57462025-04-30T06:42:55.975ZTRACtest_snapshot_basic (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
57472025-04-30T06:42:55.975ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
57482025-04-30T06:42:55.975ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {} }
57492025-04-30T06:42:55.976ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
57502025-04-30T06:42:55.977ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "bcd3602b-8f80-4ee1-b352-2211e1ba6a4a", "content-length": "702", "date": "Wed, 30 Apr 2025 06:42:55 GMT"} })
57512025-04-30T06:42:55.977ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/0/component/rot/cfpa
57522025-04-30T06:42:55.978ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "7f9d8637-2bde-420e-9d70-afc4ffb7477a", "content-length": "727", "date": "Wed, 30 Apr 2025 06:42:55 GMT"} })
57532025-04-30T06:42:55.978ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/0/component/rot/cfpa
57542025-04-30T06:42:55.979ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
57552025-04-30T06:42:55.988ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
57562025-04-30T06:42:55.988ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
57572025-04-30T06:42:55.988ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {} }
57582025-04-30T06:42:55.991ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
57592025-04-30T06:42:55.992ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "56fce538-1182-44ef-9bbe-bfa3ce6761b9", "content-length": "729", "date": "Wed, 30 Apr 2025 06:42:55 GMT"} })
57602025-04-30T06:42:55.993ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/0/component/rot/cfpa
57612025-04-30T06:42:55.993ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
57622025-04-30T06:42:55.993ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {} }
57632025-04-30T06:42:55.994ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
57642025-04-30T06:42:55.994ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "cb8c298e-7487-422b-af85-d2d862f1737f", "content-length": "728", "date": "Wed, 30 Apr 2025 06:42:55 GMT"} })
57652025-04-30T06:42:55.994ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/1
57662025-04-30T06:42:55.996ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "d6087a05-b263-4575-b43b-643239991582", "content-length": "723", "date": "Wed, 30 Apr 2025 06:42:55 GMT"} })
57672025-04-30T06:42:55.997ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/1/component/sp/caboose?firmware_slot=0
57682025-04-30T06:42:56.005ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "8e837790-7edb-4f92-b4a2-fdb6856b468c", "content-length": "109", "date": "Wed, 30 Apr 2025 06:42:55 GMT"} })
57692025-04-30T06:42:56.006ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/1/component/sp/caboose?firmware_slot=1
57702025-04-30T06:42:56.015ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "63348cc0-9a78-4c1a-a3b9-0ed4b5be6022", "content-length": "109", "date": "Wed, 30 Apr 2025 06:42:55 GMT"} })
57712025-04-30T06:42:56.015ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/1/component/rot/caboose?firmware_slot=0
57722025-04-30T06:42:56.018ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "4f59d7a9-bfb0-4799-b94f-1cbb98b80de6", "content-length": "169", "date": "Wed, 30 Apr 2025 06:42:55 GMT"} })
57732025-04-30T06:42:56.018ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/1/component/rot/caboose?firmware_slot=1
57742025-04-30T06:42:56.020ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "8730299d-7c60-44eb-9f34-d909246cc74d", "content-length": "169", "date": "Wed, 30 Apr 2025 06:42:55 GMT"} })
57752025-04-30T06:42:56.021ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/1/component/stage0/caboose?firmware_slot=0
57762025-04-30T06:42:56.023ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "1ccfe4a7-acae-4c0f-af19-81a6d3e9071b", "content-length": "178", "date": "Wed, 30 Apr 2025 06:42:55 GMT"} })
57772025-04-30T06:42:56.024ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/1/component/stage0/caboose?firmware_slot=1
57782025-04-30T06:42:56.027ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "3bbcb032-8031-4d29-99c7-752ba8051311", "content-length": "178", "date": "Wed, 30 Apr 2025 06:42:55 GMT"} })
57792025-04-30T06:42:56.027ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/1/component/rot/cmpa
57802025-04-30T06:42:56.028ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "fe8e684a-c2db-4f20-ba21-04b0f2ef96c5", "content-length": "702", "date": "Wed, 30 Apr 2025 06:42:55 GMT"} })
57812025-04-30T06:42:56.028ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/1/component/rot/cfpa
57822025-04-30T06:42:56.029ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "c829da5b-d500-4559-98cc-b60982ea6f84", "content-length": "727", "date": "Wed, 30 Apr 2025 06:42:55 GMT"} })
57832025-04-30T06:42:56.030ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/1/component/rot/cfpa
57842025-04-30T06:42:56.031ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "800d46c4-3b1a-4bf6-88ec-1b91bb53a3bf", "content-length": "729", "date": "Wed, 30 Apr 2025 06:42:55 GMT"} })
57852025-04-30T06:42:56.031ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/1/component/rot/cfpa
57862025-04-30T06:42:56.032ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "72d93427-1542-4198-b64c-e2eb840ec3f2", "content-length": "728", "date": "Wed, 30 Apr 2025 06:42:55 GMT"} })
57872025-04-30T06:42:56.032ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
57882025-04-30T06:42:56.032ZTRACtest_snapshot_basic (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
57892025-04-30T06:42:56.032ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
57902025-04-30T06:42:56.032ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
57912025-04-30T06:42:56.034ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
57922025-04-30T06:42:56.035ZDEBGtest_snapshot_basic (results-sink): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
57932025-04-30T06:42:56.036ZDEBGtest_snapshot_basic (results-sink-cluster): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
57942025-04-30T06:42:56.036ZDEBGtest_snapshot_basic (clickhouse-client): unrolling 1 total samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 6ad06c6b-9a6e-4ed1-b3f4-5d2e9da3f8a7
57952025-04-30T06:42:56.037ZTRACtest_snapshot_basic (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 6ad06c6b-9a6e-4ed1-b3f4-5d2e9da3f8a7 n_columns = 4 n_rows = 2 sql = INSERT INTO oximeter.fields_string FORMAT Native
57962025-04-30T06:42:56.041ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
57972025-04-30T06:42:56.047ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
57982025-04-30T06:42:56.047ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
57992025-04-30T06:42:56.047ZTRACtest_snapshot_basic (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
58002025-04-30T06:42:56.047ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
58012025-04-30T06:42:56.047ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
58022025-04-30T06:42:56.047ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
58032025-04-30T06:42:56.049ZDEBGtest_snapshot_basic (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 6ad06c6b-9a6e-4ed1-b3f4-5d2e9da3f8a7 n_rows = 2 table_name = fields_string
58042025-04-30T06:42:56.049ZTRACtest_snapshot_basic (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 6ad06c6b-9a6e-4ed1-b3f4-5d2e9da3f8a7 n_columns = 4 n_rows = 1 sql = INSERT INTO oximeter.measurements_i64 FORMAT Native
58052025-04-30T06:42:56.053ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
58062025-04-30T06:42:56.058ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
58072025-04-30T06:42:56.058ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
58082025-04-30T06:42:56.058ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
58092025-04-30T06:42:56.059ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
58102025-04-30T06:42:56.062ZDEBGtest_snapshot_basic: attempt to fetch latest inventory collection result = Ok("found")
58112025-04-30T06:42:56.068ZDEBGtest_snapshot_basic (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 6ad06c6b-9a6e-4ed1-b3f4-5d2e9da3f8a7 n_rows = 1 table_name = measurements_i64
58122025-04-30T06:42:56.068ZTRACtest_snapshot_basic (results-sink): successfully inserted samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
58132025-04-30T06:42:56.123ZDEBGtest_snapshot_basic (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:41923
58142025-04-30T06:42:56.123ZDEBGtest_snapshot_basic (BackgroundTasks): client request SledAgent = http://[::1]:41923 background_task = inventory_collection body = None method = GET uri = http://[::1]:41923/inventory
58152025-04-30T06:42:56.124ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
58162025-04-30T06:42:56.124ZTRACtest_snapshot_basic (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
58172025-04-30T06:42:56.124ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
58182025-04-30T06:42:56.124ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {} }
58192025-04-30T06:42:56.125ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
58202025-04-30T06:42:56.126ZINFOtest_snapshot_basic (dropshot): accepted connection local_addr = [::1]:41923 remote_addr = [::1]:51181 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
58212025-04-30T06:42:56.130ZTRACtest_snapshot_basic (dropshot): incoming request local_addr = [::1]:41923 method = GET remote_addr = [::1]:51181 req_id = 5e8026f3-5953-4b84-aef5-8291ec661638 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
58222025-04-30T06:42:56.131ZINFOtest_snapshot_basic (dropshot): request completed latency_us = 1680 local_addr = [::1]:41923 method = GET remote_addr = [::1]:51181 req_id = 5e8026f3-5953-4b84-aef5-8291ec661638 response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
58232025-04-30T06:42:56.133ZDEBGtest_snapshot_basic (BackgroundTasks): client response SledAgent = http://[::1]:41923 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:41923/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "5e8026f3-5953-4b84-aef5-8291ec661638", "content-length": "3141", "date": "Wed, 30 Apr 2025 06:42:55 GMT"} })
58242025-04-30T06:42:56.134ZDEBGtest_snapshot_basic (BackgroundTasks): begin collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
58252025-04-30T06:42:56.134ZDEBGtest_snapshot_basic (BackgroundTasks): end collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
58262025-04-30T06:42:56.134ZDEBGtest_snapshot_basic (BackgroundTasks): finished collection background_task = inventory_collection
58272025-04-30T06:42:56.136ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
58282025-04-30T06:42:56.136ZTRACtest_snapshot_basic (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
58292025-04-30T06:42:56.136ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
58302025-04-30T06:42:56.136ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
58312025-04-30T06:42:56.137ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
58322025-04-30T06:42:56.139ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
58332025-04-30T06:42:56.146ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
58342025-04-30T06:42:56.146ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
58352025-04-30T06:42:56.146ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {} }
58362025-04-30T06:42:56.146ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
58372025-04-30T06:42:56.147ZWARNtest_snapshot_basic (results-sink-cluster): failed to insert some results into metric DB collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 error = Failed to check out connection to database: No backends found for this service file = oximeter/collector/src/results_sink.rs:92
58382025-04-30T06:42:56.153ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
58392025-04-30T06:42:56.153ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {} }
58402025-04-30T06:42:56.154ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
58412025-04-30T06:42:56.156ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
58422025-04-30T06:42:56.162ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
58432025-04-30T06:42:56.162ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
58442025-04-30T06:42:56.162ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
58452025-04-30T06:42:56.168ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
58462025-04-30T06:42:56.207ZDEBGtest_snapshot_basic: attempt to fetch latest inventory collection result = Ok("found")
58472025-04-30T06:42:56.211ZWARNtest_snapshot_basic (datastore): Non-retryable transaction failure err = DatabaseError(UniqueViolation, "duplicate key value violates unique constraint \\"inv_caboose_pkey\\"") transaction = inventory_insert_collection
58482025-04-30T06:42:56.212ZWARNtest_snapshot_basic (BackgroundTasks): inventory collection failed background_task = inventory_collection error = failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint "inv_caboose_pkey"
58492025-04-30T06:42:56.212ZDEBGtest_snapshot_basic (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 461.464047ms iteration = 15 status = {"error":"failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint \\"inv_caboose_pkey\\""}
58502025-04-30T06:42:56.212ZDEBGtest_snapshot_basic (BackgroundTasks): activating background_task = inventory_collection iteration = 16 reason = Signaled
58512025-04-30T06:42:56.212ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
58522025-04-30T06:42:56.213ZTRACtest_snapshot_basic (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
58532025-04-30T06:42:56.213ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
58542025-04-30T06:42:56.213ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
58552025-04-30T06:42:56.214ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
58562025-04-30T06:42:56.219ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
58572025-04-30T06:42:56.229ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
58582025-04-30T06:42:56.229ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
58592025-04-30T06:42:56.229ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
58602025-04-30T06:42:56.230ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
58612025-04-30T06:42:56.233ZDEBGtest_snapshot_basic (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = [(d1ad44e5-18dd-4193-a70e-bf9042f0e0e1, 0)]
58622025-04-30T06:42:56.234ZTRACtest_snapshot_basic (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
58632025-04-30T06:42:56.235ZTRACtest_snapshot_basic (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: 6e 2e 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 n............_mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:59368 req_id = 10106233-421d-4110-8475-830b46f4cdf5
58642025-04-30T06:42:56.235ZDEBGtest_snapshot_basic (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 28206,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:59368 req_id = 10106233-421d-4110-8475-830b46f4cdf5
58652025-04-30T06:42:56.235ZDEBGtest_snapshot_basic (store): query key key = _mgs._tcp
58662025-04-30T06:42:56.236ZDEBGtest_snapshot_basic (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
58672025-04-30T06:42:56.236ZDEBGtest_snapshot_basic (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:59368 query = LowerQuery { name: LowerName(Name("_mgs._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 41502, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = 10106233-421d-4110-8475-830b46f4cdf5
58682025-04-30T06:42:56.237ZDEBGtest_snapshot_basic (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 41502, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 4241, tv_nsec: 36489064 } })
58692025-04-30T06:42:56.237ZTRACtest_snapshot_basic (DnsResolver): lookup_service_targets: looking up SRV target
58702025-04-30T06:42:56.238ZTRACtest_snapshot_basic (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: 39 90 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e 9...........-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:58638 req_id = 1294ace7-512c-49ee-a6ee-e8a9a36bd664
58712025-04-30T06:42:56.238ZDEBGtest_snapshot_basic (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 14736,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:58638 req_id = 1294ace7-512c-49ee-a6ee-e8a9a36bd664
58722025-04-30T06:42:56.238ZDEBGtest_snapshot_basic (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
58732025-04-30T06:42:56.238ZDEBGtest_snapshot_basic (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:58638 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = 1294ace7-512c-49ee-a6ee-e8a9a36bd664
58742025-04-30T06:42:56.305ZTRACtest_snapshot_basic (DnsResolver): lookup_all_socket_v6 srv dns_name = _clickhouse-admin-keeper._tcp.control-plane.oxide.internal
58752025-04-30T06:42:56.305ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
58762025-04-30T06:42:56.305ZTRACtest_snapshot_basic (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
58772025-04-30T06:42:56.305ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
58782025-04-30T06:42:56.305ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {} }
58792025-04-30T06:42:56.306ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
58802025-04-30T06:42:56.306ZTRACtest_snapshot_basic (dns): buffer buffer = Length: 87 (0x57) bytes\n0000: d3 58 01 00 00 01 00 00 00 00 00 01 18 5f 63 6c .X..........._cl\n0010: 69 63 6b 68 6f 75 73 65 2d 61 64 6d 69 6e 2d 6b ickhouse-admin-k\n0020: 65 65 70 65 72 04 5f 74 63 70 0d 63 6f 6e 74 72 eeper._tcp.contr\n0030: 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 69 ol-plane.oxide.i\n0040: 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 04 nternal..!....).\n0050: d0 00 00 00 00 00 00 ....... kind = dns peer_addr = [::1]:52941 req_id = 98fd981c-3322-4019-9ed1-3a3057a475f7
58812025-04-30T06:42:56.307ZDEBGtest_snapshot_basic (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 54104,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 24,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 97,\n 100,\n 109,\n 105,\n 110,\n 45,\n 107,\n 101,\n 101,\n 112,\n 101,\n 114,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:52941 req_id = 98fd981c-3322-4019-9ed1-3a3057a475f7
58822025-04-30T06:42:56.307ZDEBGtest_snapshot_basic (store): query key key = _clickhouse-admin-keeper._tcp
58832025-04-30T06:42:56.307ZERROtest_snapshot_basic (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 54104, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } NXDOMAIN: no records found for name: "_clickhouse-admin-keeper._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:52941 req_id = 98fd981c-3322-4019-9ed1-3a3057a475f7
58842025-04-30T06:42:56.308ZDEBGtest_snapshot_basic (BackgroundTasks): begin collection background_task = inventory_collection
58852025-04-30T06:42:56.308ZDEBGtest_snapshot_basic (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:41502
58862025-04-30T06:42:56.308ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/ignition
58872025-04-30T06:42:56.309ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
58882025-04-30T06:42:56.315ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
58892025-04-30T06:42:56.315ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
58902025-04-30T06:42:56.315ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {} }
58912025-04-30T06:42:56.316ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
58922025-04-30T06:42:56.318ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "03b8abde-ebdb-47c3-822f-00d48f32ba24", "content-length": "835", "date": "Wed, 30 Apr 2025 06:42:55 GMT"} })
58932025-04-30T06:42:56.318ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/0
58942025-04-30T06:42:56.319ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
58952025-04-30T06:42:56.319ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {} }
58962025-04-30T06:42:56.319ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
58972025-04-30T06:42:56.322ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "614b5469-fb0e-4980-aac1-5efc003b8778", "content-length": "734", "date": "Wed, 30 Apr 2025 06:42:55 GMT"} })
58982025-04-30T06:42:56.322ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/0/component/sp/caboose?firmware_slot=0
58992025-04-30T06:42:56.340ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "1ad31d6b-a28e-417e-90f3-97d0d2ad4b1e", "content-length": "111", "date": "Wed, 30 Apr 2025 06:42:56 GMT"} })
59002025-04-30T06:42:56.340ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/0/component/sp/caboose?firmware_slot=1
59012025-04-30T06:42:56.343ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "c6d98a91-fdc5-45f2-ad8c-2d77a8f221ef", "content-length": "111", "date": "Wed, 30 Apr 2025 06:42:56 GMT"} })
59022025-04-30T06:42:56.344ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/0/component/rot/caboose?firmware_slot=0
59032025-04-30T06:42:56.349ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "acf97add-39c7-4c72-a8b2-894a33a452d1", "content-length": "170", "date": "Wed, 30 Apr 2025 06:42:56 GMT"} })
59042025-04-30T06:42:56.349ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/0/component/rot/caboose?firmware_slot=1
59052025-04-30T06:42:56.353ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "9c26e3b0-2809-4927-8013-4881c3096599", "content-length": "170", "date": "Wed, 30 Apr 2025 06:42:56 GMT"} })
59062025-04-30T06:42:56.353ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/0/component/stage0/caboose?firmware_slot=0
59072025-04-30T06:42:56.355ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "2d2f6ee1-fede-458f-adb2-46c2743dc2cc", "content-length": "179", "date": "Wed, 30 Apr 2025 06:42:56 GMT"} })
59082025-04-30T06:42:56.356ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/0/component/stage0/caboose?firmware_slot=1
59092025-04-30T06:42:56.359ZDEBGtest_snapshot_basic: attempt to fetch latest inventory collection result = Ok("found")
59102025-04-30T06:42:56.360ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "c52c9154-ad15-4a41-9f3c-2eff6986c270", "content-length": "179", "date": "Wed, 30 Apr 2025 06:42:56 GMT"} })
59112025-04-30T06:42:56.362ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/0/component/rot/cmpa
59122025-04-30T06:42:56.362ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "f66d3928-d85f-4b73-bba2-3550fb8a203f", "content-length": "702", "date": "Wed, 30 Apr 2025 06:42:56 GMT"} })
59132025-04-30T06:42:56.362ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/0/component/rot/cfpa
59142025-04-30T06:42:56.365ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "b59f75f5-1107-4df8-ba49-124ee2d74f04", "content-length": "727", "date": "Wed, 30 Apr 2025 06:42:56 GMT"} })
59152025-04-30T06:42:56.365ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/0/component/rot/cfpa
59162025-04-30T06:42:56.365ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "756af780-b2a6-42bb-8a64-4ef48471197f", "content-length": "729", "date": "Wed, 30 Apr 2025 06:42:56 GMT"} })
59172025-04-30T06:42:56.365ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/0/component/rot/cfpa
59182025-04-30T06:42:56.365ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "187f8c68-0150-4121-86fd-6efc2ddd9491", "content-length": "728", "date": "Wed, 30 Apr 2025 06:42:56 GMT"} })
59192025-04-30T06:42:56.365ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/1
59202025-04-30T06:42:56.368ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "4b573007-cbe1-4eb7-a385-196951c2c022", "content-length": "734", "date": "Wed, 30 Apr 2025 06:42:56 GMT"} })
59212025-04-30T06:42:56.368ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/1/component/sp/caboose?firmware_slot=0
59222025-04-30T06:42:56.371ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "040e7578-7d3e-4c19-a601-d6252ed4f09f", "content-length": "111", "date": "Wed, 30 Apr 2025 06:42:56 GMT"} })
59232025-04-30T06:42:56.371ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/1/component/sp/caboose?firmware_slot=1
59242025-04-30T06:42:56.372ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "4301f8e1-5dc1-467a-9c67-09e3c7914efa", "content-length": "111", "date": "Wed, 30 Apr 2025 06:42:56 GMT"} })
59252025-04-30T06:42:56.373ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/1/component/rot/caboose?firmware_slot=0
59262025-04-30T06:42:56.376ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "ec3c924f-f638-4caf-a069-38e25f9e9c99", "content-length": "170", "date": "Wed, 30 Apr 2025 06:42:56 GMT"} })
59272025-04-30T06:42:56.376ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/1/component/rot/caboose?firmware_slot=1
59282025-04-30T06:42:56.377ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "e9d1b6b2-9cda-4f16-8174-5db13f72a2da", "content-length": "170", "date": "Wed, 30 Apr 2025 06:42:56 GMT"} })
59292025-04-30T06:42:56.377ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/1/component/stage0/caboose?firmware_slot=0
59302025-04-30T06:42:56.380ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "a5840b69-f612-4ceb-ab9d-a7110d6e052e", "content-length": "179", "date": "Wed, 30 Apr 2025 06:42:56 GMT"} })
59312025-04-30T06:42:56.380ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/1/component/stage0/caboose?firmware_slot=1
59322025-04-30T06:42:56.382ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "9e1020d5-d8fe-418b-9544-4827c5e41ddb", "content-length": "179", "date": "Wed, 30 Apr 2025 06:42:56 GMT"} })
59332025-04-30T06:42:56.382ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/1/component/rot/cmpa
59342025-04-30T06:42:56.383ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "c4a6cd41-5984-48a0-8001-23b0057bd5f4", "content-length": "702", "date": "Wed, 30 Apr 2025 06:42:56 GMT"} })
59352025-04-30T06:42:56.383ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/1/component/rot/cfpa
59362025-04-30T06:42:56.385ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "91986e25-e727-412f-af06-a801ca8ba13e", "content-length": "727", "date": "Wed, 30 Apr 2025 06:42:56 GMT"} })
59372025-04-30T06:42:56.385ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/1/component/rot/cfpa
59382025-04-30T06:42:56.389ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "54445806-42f3-454a-9749-cb66582b448c", "content-length": "729", "date": "Wed, 30 Apr 2025 06:42:56 GMT"} })
59392025-04-30T06:42:56.390ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/1/component/rot/cfpa
59402025-04-30T06:42:56.391ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "2a93d2e8-f374-4670-ad68-baaede460ed3", "content-length": "728", "date": "Wed, 30 Apr 2025 06:42:56 GMT"} })
59412025-04-30T06:42:56.391ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/0
59422025-04-30T06:42:56.392ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "0bd44ebd-f89c-4ec7-ab27-95be88cfe0a3", "content-length": "723", "date": "Wed, 30 Apr 2025 06:42:56 GMT"} })
59432025-04-30T06:42:56.392ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/0/component/sp/caboose?firmware_slot=0
59442025-04-30T06:42:56.394ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "3ccea186-f490-41a9-bd01-9cf13d2abbed", "content-length": "109", "date": "Wed, 30 Apr 2025 06:42:56 GMT"} })
59452025-04-30T06:42:56.394ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/0/component/sp/caboose?firmware_slot=1
59462025-04-30T06:42:56.396ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "4110acf5-fa30-4403-ad5e-1151bd9bf894", "content-length": "109", "date": "Wed, 30 Apr 2025 06:42:56 GMT"} })
59472025-04-30T06:42:56.396ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/0/component/rot/caboose?firmware_slot=0
59482025-04-30T06:42:56.398ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "8b21d215-eba7-4d69-8a2b-f50eb94a87cd", "content-length": "169", "date": "Wed, 30 Apr 2025 06:42:56 GMT"} })
59492025-04-30T06:42:56.398ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/0/component/rot/caboose?firmware_slot=1
59502025-04-30T06:42:56.399ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "5bf4f16e-83a8-417c-a7a1-d31edc8248bf", "content-length": "169", "date": "Wed, 30 Apr 2025 06:42:56 GMT"} })
59512025-04-30T06:42:56.399ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/0/component/stage0/caboose?firmware_slot=0
59522025-04-30T06:42:56.401ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "a08e04b0-4c61-4e02-b0d6-f62cbfe9352a", "content-length": "178", "date": "Wed, 30 Apr 2025 06:42:56 GMT"} })
59532025-04-30T06:42:56.401ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/0/component/stage0/caboose?firmware_slot=1
59542025-04-30T06:42:56.403ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "1512d84d-ffe2-4e9f-9935-204812b71a91", "content-length": "178", "date": "Wed, 30 Apr 2025 06:42:56 GMT"} })
59552025-04-30T06:42:56.403ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/0/component/rot/cmpa
59562025-04-30T06:42:56.404ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "2fa0d8bd-c9bc-406d-8a2b-1ba5d1b2cb25", "content-length": "702", "date": "Wed, 30 Apr 2025 06:42:56 GMT"} })
59572025-04-30T06:42:56.405ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/0/component/rot/cfpa
59582025-04-30T06:42:56.406ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "ff8d65da-aca0-4dda-a079-e68a98f2f300", "content-length": "727", "date": "Wed, 30 Apr 2025 06:42:56 GMT"} })
59592025-04-30T06:42:56.406ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/0/component/rot/cfpa
59602025-04-30T06:42:56.407ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "1345079c-4ec8-4aca-bac4-7b71c100cc0a", "content-length": "729", "date": "Wed, 30 Apr 2025 06:42:56 GMT"} })
59612025-04-30T06:42:56.407ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/0/component/rot/cfpa
59622025-04-30T06:42:56.408ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "2361bcab-dc2c-420f-af34-00e6a5ede6d4", "content-length": "728", "date": "Wed, 30 Apr 2025 06:42:56 GMT"} })
59632025-04-30T06:42:56.408ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/1
59642025-04-30T06:42:56.409ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "0a7f5d94-4950-4bdc-b9a5-48465c69cbd7", "content-length": "723", "date": "Wed, 30 Apr 2025 06:42:56 GMT"} })
59652025-04-30T06:42:56.409ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/1/component/sp/caboose?firmware_slot=0
59662025-04-30T06:42:56.411ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
59672025-04-30T06:42:56.411ZTRACtest_snapshot_basic (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
59682025-04-30T06:42:56.411ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
59692025-04-30T06:42:56.411ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {} }
59702025-04-30T06:42:56.411ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
59712025-04-30T06:42:56.413ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "a7376013-6864-4e0c-a31a-93207696bbc9", "content-length": "109", "date": "Wed, 30 Apr 2025 06:42:56 GMT"} })
59722025-04-30T06:42:56.413ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/1/component/sp/caboose?firmware_slot=1
59732025-04-30T06:42:56.413ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
59742025-04-30T06:42:56.420ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
59752025-04-30T06:42:56.420ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
59762025-04-30T06:42:56.420ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {} }
59772025-04-30T06:42:56.421ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
59782025-04-30T06:42:56.424ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
59792025-04-30T06:42:56.424ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {} }
59802025-04-30T06:42:56.424ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
59812025-04-30T06:42:56.426ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "ba22e5cf-cdf1-45e0-ae66-aa8ffbfd28a5", "content-length": "109", "date": "Wed, 30 Apr 2025 06:42:56 GMT"} })
59822025-04-30T06:42:56.426ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/1/component/rot/caboose?firmware_slot=0
59832025-04-30T06:42:56.430ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "39786fdc-c2f3-4b8c-b40e-53b6ba238d69", "content-length": "169", "date": "Wed, 30 Apr 2025 06:42:56 GMT"} })
59842025-04-30T06:42:56.430ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/1/component/rot/caboose?firmware_slot=1
59852025-04-30T06:42:56.433ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "5e6dffd3-a124-40cf-827f-ca1c91cd6ad0", "content-length": "169", "date": "Wed, 30 Apr 2025 06:42:56 GMT"} })
59862025-04-30T06:42:56.433ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/1/component/stage0/caboose?firmware_slot=0
59872025-04-30T06:42:56.441ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "a48f29d6-d0b5-4d0d-b636-01520af7a6f8", "content-length": "178", "date": "Wed, 30 Apr 2025 06:42:56 GMT"} })
59882025-04-30T06:42:56.441ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/1/component/stage0/caboose?firmware_slot=1
59892025-04-30T06:42:56.446ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "44894d85-cbd0-4620-9aae-123b3cb080ea", "content-length": "178", "date": "Wed, 30 Apr 2025 06:42:56 GMT"} })
59902025-04-30T06:42:56.447ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/1/component/rot/cmpa
59912025-04-30T06:42:56.450ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "a86ab714-57d1-4ecd-ba5b-f58b2b74a8d0", "content-length": "702", "date": "Wed, 30 Apr 2025 06:42:56 GMT"} })
59922025-04-30T06:42:56.450ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/1/component/rot/cfpa
59932025-04-30T06:42:56.451ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "9dd01c96-8d95-46e6-bfc0-a7940380f433", "content-length": "727", "date": "Wed, 30 Apr 2025 06:42:56 GMT"} })
59942025-04-30T06:42:56.452ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/1/component/rot/cfpa
59952025-04-30T06:42:56.453ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "7ed0a58d-7dd2-4858-a714-0103375301a9", "content-length": "729", "date": "Wed, 30 Apr 2025 06:42:56 GMT"} })
59962025-04-30T06:42:56.453ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/1/component/rot/cfpa
59972025-04-30T06:42:56.455ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "f6c4cfc5-d09e-4274-87b7-591708b9f623", "content-length": "728", "date": "Wed, 30 Apr 2025 06:42:56 GMT"} })
59982025-04-30T06:42:56.455ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
59992025-04-30T06:42:56.456ZTRACtest_snapshot_basic (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
60002025-04-30T06:42:56.456ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
60012025-04-30T06:42:56.456ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
60022025-04-30T06:42:56.456ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
60032025-04-30T06:42:56.457ZDEBGtest_snapshot_basic: attempt to fetch latest inventory collection result = Ok("found")
60042025-04-30T06:42:56.460ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
60052025-04-30T06:42:56.472ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
60062025-04-30T06:42:56.472ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
60072025-04-30T06:42:56.472ZTRACtest_snapshot_basic (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
60082025-04-30T06:42:56.472ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
60092025-04-30T06:42:56.473ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
60102025-04-30T06:42:56.474ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
60112025-04-30T06:42:56.477ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
60122025-04-30T06:42:56.482ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
60132025-04-30T06:42:56.482ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
60142025-04-30T06:42:56.482ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
60152025-04-30T06:42:56.483ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
60162025-04-30T06:42:56.546ZDEBGtest_snapshot_basic (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:41923
60172025-04-30T06:42:56.546ZDEBGtest_snapshot_basic (BackgroundTasks): client request SledAgent = http://[::1]:41923 background_task = inventory_collection body = None method = GET uri = http://[::1]:41923/inventory
60182025-04-30T06:42:56.547ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
60192025-04-30T06:42:56.547ZTRACtest_snapshot_basic (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
60202025-04-30T06:42:56.547ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
60212025-04-30T06:42:56.547ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {} }
60222025-04-30T06:42:56.548ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
60232025-04-30T06:42:56.548ZINFOtest_snapshot_basic (dropshot): accepted connection local_addr = [::1]:41923 remote_addr = [::1]:33656 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
60242025-04-30T06:42:56.550ZTRACtest_snapshot_basic (dropshot): incoming request local_addr = [::1]:41923 method = GET remote_addr = [::1]:33656 req_id = 3b3ecfc3-cff7-4c8d-a281-c5ef7a413949 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
60252025-04-30T06:42:56.550ZINFOtest_snapshot_basic (dropshot): request completed latency_us = 553 local_addr = [::1]:41923 method = GET remote_addr = [::1]:33656 req_id = 3b3ecfc3-cff7-4c8d-a281-c5ef7a413949 response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
60262025-04-30T06:42:56.551ZDEBGtest_snapshot_basic (BackgroundTasks): client response SledAgent = http://[::1]:41923 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:41923/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "3b3ecfc3-cff7-4c8d-a281-c5ef7a413949", "content-length": "3141", "date": "Wed, 30 Apr 2025 06:42:56 GMT"} })
60272025-04-30T06:42:56.553ZDEBGtest_snapshot_basic (BackgroundTasks): begin collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
60282025-04-30T06:42:56.553ZDEBGtest_snapshot_basic (BackgroundTasks): end collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
60292025-04-30T06:42:56.553ZDEBGtest_snapshot_basic (BackgroundTasks): finished collection background_task = inventory_collection
60302025-04-30T06:42:56.556ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
60312025-04-30T06:42:56.556ZTRACtest_snapshot_basic (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
60322025-04-30T06:42:56.556ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
60332025-04-30T06:42:56.556ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
60342025-04-30T06:42:56.556ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
60352025-04-30T06:42:56.557ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
60362025-04-30T06:42:56.562ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
60372025-04-30T06:42:56.562ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
60382025-04-30T06:42:56.562ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {} }
60392025-04-30T06:42:56.562ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
60402025-04-30T06:42:56.568ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
60412025-04-30T06:42:56.572ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
60422025-04-30T06:42:56.572ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
60432025-04-30T06:42:56.572ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
60442025-04-30T06:42:56.573ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
60452025-04-30T06:42:56.575ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
60462025-04-30T06:42:56.575ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {} }
60472025-04-30T06:42:56.576ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
60482025-04-30T06:42:56.633ZWARNtest_snapshot_basic (datastore): Non-retryable transaction failure err = DatabaseError(UniqueViolation, "duplicate key value violates unique constraint \\"inv_caboose_pkey\\"") transaction = inventory_insert_collection
60492025-04-30T06:42:56.633ZWARNtest_snapshot_basic (BackgroundTasks): inventory collection failed background_task = inventory_collection error = failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint "inv_caboose_pkey"
60502025-04-30T06:42:56.633ZDEBGtest_snapshot_basic (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 421.214803ms iteration = 16 status = {"error":"failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint \\"inv_caboose_pkey\\""}
60512025-04-30T06:42:56.633ZDEBGtest_snapshot_basic (BackgroundTasks): activating background_task = inventory_collection iteration = 17 reason = Signaled
60522025-04-30T06:42:56.633ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
60532025-04-30T06:42:56.633ZTRACtest_snapshot_basic (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
60542025-04-30T06:42:56.633ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
60552025-04-30T06:42:56.633ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
60562025-04-30T06:42:56.635ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
60572025-04-30T06:42:56.638ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
60582025-04-30T06:42:56.642ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
60592025-04-30T06:42:56.642ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
60602025-04-30T06:42:56.642ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
60612025-04-30T06:42:56.643ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
60622025-04-30T06:42:56.646ZDEBGtest_snapshot_basic: attempt to fetch latest inventory collection result = Ok("found")
60632025-04-30T06:42:56.648ZDEBGtest_snapshot_basic (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = [(d1ad44e5-18dd-4193-a70e-bf9042f0e0e1, 0)]
60642025-04-30T06:42:56.648ZTRACtest_snapshot_basic (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
60652025-04-30T06:42:56.649ZTRACtest_snapshot_basic (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: bf 1d 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 ............._mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:52883 req_id = ae8c00ac-c0f7-4678-bee0-f57b4e37405e
60662025-04-30T06:42:56.649ZDEBGtest_snapshot_basic (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 48925,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:52883 req_id = ae8c00ac-c0f7-4678-bee0-f57b4e37405e
60672025-04-30T06:42:56.650ZDEBGtest_snapshot_basic (store): query key key = _mgs._tcp
60682025-04-30T06:42:56.650ZDEBGtest_snapshot_basic (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
60692025-04-30T06:42:56.650ZDEBGtest_snapshot_basic (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:52883 query = LowerQuery { name: LowerName(Name("_mgs._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 41502, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = ae8c00ac-c0f7-4678-bee0-f57b4e37405e
60702025-04-30T06:42:56.650ZDEBGtest_snapshot_basic (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 41502, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 4241, tv_nsec: 449805721 } })
60712025-04-30T06:42:56.650ZTRACtest_snapshot_basic (DnsResolver): lookup_service_targets: looking up SRV target
60722025-04-30T06:42:56.651ZTRACtest_snapshot_basic (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: 8c 6d 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e .m..........-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:60375 req_id = 1dfdf5c7-0b38-47ba-88b7-d6bcf798be12
60732025-04-30T06:42:56.651ZDEBGtest_snapshot_basic (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 35949,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:60375 req_id = 1dfdf5c7-0b38-47ba-88b7-d6bcf798be12
60742025-04-30T06:42:56.651ZDEBGtest_snapshot_basic (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
60752025-04-30T06:42:56.651ZDEBGtest_snapshot_basic (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:60375 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = 1dfdf5c7-0b38-47ba-88b7-d6bcf798be12
60762025-04-30T06:42:56.715ZTRACtest_snapshot_basic (DnsResolver): lookup_all_socket_v6 srv dns_name = _clickhouse-admin-keeper._tcp.control-plane.oxide.internal
60772025-04-30T06:42:56.716ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
60782025-04-30T06:42:56.716ZTRACtest_snapshot_basic (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
60792025-04-30T06:42:56.716ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
60802025-04-30T06:42:56.716ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {} }
60812025-04-30T06:42:56.717ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
60822025-04-30T06:42:56.717ZTRACtest_snapshot_basic (dns): buffer buffer = Length: 87 (0x57) bytes\n0000: b4 ef 01 00 00 01 00 00 00 00 00 01 18 5f 63 6c ............._cl\n0010: 69 63 6b 68 6f 75 73 65 2d 61 64 6d 69 6e 2d 6b ickhouse-admin-k\n0020: 65 65 70 65 72 04 5f 74 63 70 0d 63 6f 6e 74 72 eeper._tcp.contr\n0030: 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 69 ol-plane.oxide.i\n0040: 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 04 nternal..!....).\n0050: d0 00 00 00 00 00 00 ....... kind = dns peer_addr = [::1]:55233 req_id = 6bdd875b-a887-45bb-8673-355736dcecb1
60832025-04-30T06:42:56.717ZDEBGtest_snapshot_basic (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 46319,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 24,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 97,\n 100,\n 109,\n 105,\n 110,\n 45,\n 107,\n 101,\n 101,\n 112,\n 101,\n 114,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:55233 req_id = 6bdd875b-a887-45bb-8673-355736dcecb1
60842025-04-30T06:42:56.718ZDEBGtest_snapshot_basic (store): query key key = _clickhouse-admin-keeper._tcp
60852025-04-30T06:42:56.718ZERROtest_snapshot_basic (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 46319, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } NXDOMAIN: no records found for name: "_clickhouse-admin-keeper._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:55233 req_id = 6bdd875b-a887-45bb-8673-355736dcecb1
60862025-04-30T06:42:56.718ZDEBGtest_snapshot_basic (BackgroundTasks): begin collection background_task = inventory_collection
60872025-04-30T06:42:56.718ZDEBGtest_snapshot_basic (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:41502
60882025-04-30T06:42:56.718ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/ignition
60892025-04-30T06:42:56.720ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "324039fe-0ef1-4a0c-b9fc-9653b3f56876", "content-length": "835", "date": "Wed, 30 Apr 2025 06:42:56 GMT"} })
60902025-04-30T06:42:56.720ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/0
60912025-04-30T06:42:56.720ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
60922025-04-30T06:42:56.725ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
60932025-04-30T06:42:56.725ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
60942025-04-30T06:42:56.725ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {} }
60952025-04-30T06:42:56.725ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
60962025-04-30T06:42:56.728ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "cd0e17d2-3931-419c-9b95-65808925e8f2", "content-length": "734", "date": "Wed, 30 Apr 2025 06:42:56 GMT"} })
60972025-04-30T06:42:56.728ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/0/component/sp/caboose?firmware_slot=0
60982025-04-30T06:42:56.728ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
60992025-04-30T06:42:56.728ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {} }
61002025-04-30T06:42:56.729ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
61012025-04-30T06:42:56.731ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "1e9237a4-7e6f-4bbc-977d-ca5d06331d8a", "content-length": "111", "date": "Wed, 30 Apr 2025 06:42:56 GMT"} })
61022025-04-30T06:42:56.731ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/0/component/sp/caboose?firmware_slot=1
61032025-04-30T06:42:56.738ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "0803299e-f3dd-465c-8a20-224e2e3b6b72", "content-length": "111", "date": "Wed, 30 Apr 2025 06:42:56 GMT"} })
61042025-04-30T06:42:56.739ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/0/component/rot/caboose?firmware_slot=0
61052025-04-30T06:42:56.741ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "4f466c08-52e9-46f0-89a3-9d66cf7340df", "content-length": "170", "date": "Wed, 30 Apr 2025 06:42:56 GMT"} })
61062025-04-30T06:42:56.741ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/0/component/rot/caboose?firmware_slot=1
61072025-04-30T06:42:56.744ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "d44e3b78-b2da-41fe-9324-cb91297623d4", "content-length": "170", "date": "Wed, 30 Apr 2025 06:42:56 GMT"} })
61082025-04-30T06:42:56.744ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/0/component/stage0/caboose?firmware_slot=0
61092025-04-30T06:42:56.746ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "4d6c6225-c0a4-4dd2-aa85-24fe938f1e64", "content-length": "179", "date": "Wed, 30 Apr 2025 06:42:56 GMT"} })
61102025-04-30T06:42:56.746ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/0/component/stage0/caboose?firmware_slot=1
61112025-04-30T06:42:56.748ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "8e22509c-6379-481b-a4b7-5783a8d03914", "content-length": "179", "date": "Wed, 30 Apr 2025 06:42:56 GMT"} })
61122025-04-30T06:42:56.748ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/0/component/rot/cmpa
61132025-04-30T06:42:56.749ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "cf27150b-2c97-4f61-80a2-54c538ac7578", "content-length": "702", "date": "Wed, 30 Apr 2025 06:42:56 GMT"} })
61142025-04-30T06:42:56.749ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/0/component/rot/cfpa
61152025-04-30T06:42:56.750ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "297a6a9f-6f69-414e-a982-89672c365ecf", "content-length": "727", "date": "Wed, 30 Apr 2025 06:42:56 GMT"} })
61162025-04-30T06:42:56.750ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/0/component/rot/cfpa
61172025-04-30T06:42:56.751ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "7a1cd13d-6208-4adf-b288-ae862197b0c8", "content-length": "729", "date": "Wed, 30 Apr 2025 06:42:56 GMT"} })
61182025-04-30T06:42:56.751ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/0/component/rot/cfpa
61192025-04-30T06:42:56.752ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "1dc67f18-0545-4f62-9b27-9e3afaa3ee76", "content-length": "728", "date": "Wed, 30 Apr 2025 06:42:56 GMT"} })
61202025-04-30T06:42:56.753ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/1
61212025-04-30T06:42:56.755ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "c21f85b8-ff8a-40a0-9a68-07762c766d23", "content-length": "734", "date": "Wed, 30 Apr 2025 06:42:56 GMT"} })
61222025-04-30T06:42:56.755ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/1/component/sp/caboose?firmware_slot=0
61232025-04-30T06:42:56.756ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "b76425b0-21ab-42f2-87e8-aa9063e9555b", "content-length": "111", "date": "Wed, 30 Apr 2025 06:42:56 GMT"} })
61242025-04-30T06:42:56.756ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/1/component/sp/caboose?firmware_slot=1
61252025-04-30T06:42:56.758ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "0b3c0d0f-40b4-4326-88a2-0ce9f04e1f3c", "content-length": "111", "date": "Wed, 30 Apr 2025 06:42:56 GMT"} })
61262025-04-30T06:42:56.759ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/1/component/rot/caboose?firmware_slot=0
61272025-04-30T06:42:56.761ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "e2dc5af5-4bab-4cef-a6c4-66c9ae833c14", "content-length": "170", "date": "Wed, 30 Apr 2025 06:42:56 GMT"} })
61282025-04-30T06:42:56.761ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/1/component/rot/caboose?firmware_slot=1
61292025-04-30T06:42:56.763ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "73ae6bb4-0f1a-491d-829b-0879004b2e9b", "content-length": "170", "date": "Wed, 30 Apr 2025 06:42:56 GMT"} })
61302025-04-30T06:42:56.763ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/1/component/stage0/caboose?firmware_slot=0
61312025-04-30T06:42:56.765ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "827ec544-6d06-4b9b-899a-6119720d0ad0", "content-length": "179", "date": "Wed, 30 Apr 2025 06:42:56 GMT"} })
61322025-04-30T06:42:56.765ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/1/component/stage0/caboose?firmware_slot=1
61332025-04-30T06:42:56.769ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "8f0127dc-9eea-42b7-a9ad-f84197e16406", "content-length": "179", "date": "Wed, 30 Apr 2025 06:42:56 GMT"} })
61342025-04-30T06:42:56.769ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/1/component/rot/cmpa
61352025-04-30T06:42:56.771ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "b9d8413d-fa3d-49ec-990e-f5085ca6ee09", "content-length": "702", "date": "Wed, 30 Apr 2025 06:42:56 GMT"} })
61362025-04-30T06:42:56.771ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/1/component/rot/cfpa
61372025-04-30T06:42:56.772ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "762605e4-3f8f-41b3-9b38-0f62c45dbc8f", "content-length": "727", "date": "Wed, 30 Apr 2025 06:42:56 GMT"} })
61382025-04-30T06:42:56.772ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/1/component/rot/cfpa
61392025-04-30T06:42:56.773ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "1313d5f6-63e9-4a7c-99ef-a6dd8a464b9c", "content-length": "729", "date": "Wed, 30 Apr 2025 06:42:56 GMT"} })
61402025-04-30T06:42:56.774ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/1/component/rot/cfpa
61412025-04-30T06:42:56.775ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "e61a4626-f4e6-43bd-88ba-34d073754d55", "content-length": "728", "date": "Wed, 30 Apr 2025 06:42:56 GMT"} })
61422025-04-30T06:42:56.775ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/0
61432025-04-30T06:42:56.776ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "2e1835bb-cf43-47be-8a12-3b165f6f0981", "content-length": "723", "date": "Wed, 30 Apr 2025 06:42:56 GMT"} })
61442025-04-30T06:42:56.777ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/0/component/sp/caboose?firmware_slot=0
61452025-04-30T06:42:56.817ZDEBGtest_snapshot_basic (collection-task): sent timer-based collection request to the collection task collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
61462025-04-30T06:42:56.817ZDEBGtest_snapshot_basic (collection-task): collecting from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
61472025-04-30T06:42:56.819ZTRACtest_snapshot_basic (collection-task): sent collection request to producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
61482025-04-30T06:42:56.820ZDEBGtest_snapshot_basic (collection-task): collected results from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 n_results = 1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
61492025-04-30T06:42:56.820ZTRACtest_snapshot_basic (collection-task): forwarded results to main collection loop collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
61502025-04-30T06:42:56.820ZTRACtest_snapshot_basic (collection-task): top of inner collection loop, waiting for next request collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
61512025-04-30T06:42:56.822ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "b79036cb-7d44-4f61-8d6a-1b9cbdad36b0", "content-length": "109", "date": "Wed, 30 Apr 2025 06:42:56 GMT"} })
61522025-04-30T06:42:56.822ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/0/component/sp/caboose?firmware_slot=1
61532025-04-30T06:42:56.826ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "3e67cbd9-33de-4450-a0ed-d3c1c7383a0c", "content-length": "109", "date": "Wed, 30 Apr 2025 06:42:56 GMT"} })
61542025-04-30T06:42:56.826ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/0/component/rot/caboose?firmware_slot=0
61552025-04-30T06:42:56.832ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "ba2a334d-5bd2-48e9-acb8-24847816b78e", "content-length": "169", "date": "Wed, 30 Apr 2025 06:42:56 GMT"} })
61562025-04-30T06:42:56.833ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/0/component/rot/caboose?firmware_slot=1
61572025-04-30T06:42:56.839ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "331e0d79-3080-4efc-bac3-31d39d35dcb3", "content-length": "169", "date": "Wed, 30 Apr 2025 06:42:56 GMT"} })
61582025-04-30T06:42:56.840ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/0/component/stage0/caboose?firmware_slot=0
61592025-04-30T06:42:56.847ZDEBGtest_snapshot_basic: attempt to fetch latest inventory collection result = Ok("found")
61602025-04-30T06:42:56.855ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "ce64c7ab-c6c8-469f-9ccd-e695970381a6", "content-length": "178", "date": "Wed, 30 Apr 2025 06:42:56 GMT"} })
61612025-04-30T06:42:56.856ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/0/component/stage0/caboose?firmware_slot=1
61622025-04-30T06:42:56.860ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "01e36ff8-fc2b-4e4e-9cd4-bdc8d0dab0de", "content-length": "178", "date": "Wed, 30 Apr 2025 06:42:56 GMT"} })
61632025-04-30T06:42:56.860ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/0/component/rot/cmpa
61642025-04-30T06:42:56.862ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "c0ad6259-09d8-42c2-9f33-dfd91ebb6ad2", "content-length": "702", "date": "Wed, 30 Apr 2025 06:42:56 GMT"} })
61652025-04-30T06:42:56.862ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/0/component/rot/cfpa
61662025-04-30T06:42:56.864ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "7169379b-ef9b-4f54-a929-19b7fb9f32d5", "content-length": "727", "date": "Wed, 30 Apr 2025 06:42:56 GMT"} })
61672025-04-30T06:42:56.864ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/0/component/rot/cfpa
61682025-04-30T06:42:56.866ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "32ab44c1-cf99-4b53-b70c-ec6ce81e4eb7", "content-length": "729", "date": "Wed, 30 Apr 2025 06:42:56 GMT"} })
61692025-04-30T06:42:56.867ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/0/component/rot/cfpa
61702025-04-30T06:42:56.869ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "87e936db-f649-45f6-845a-015d88776214", "content-length": "728", "date": "Wed, 30 Apr 2025 06:42:56 GMT"} })
61712025-04-30T06:42:56.869ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/1
61722025-04-30T06:42:56.871ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "43e0b8b1-2b97-49bf-9af8-245658226e66", "content-length": "723", "date": "Wed, 30 Apr 2025 06:42:56 GMT"} })
61732025-04-30T06:42:56.871ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/1/component/sp/caboose?firmware_slot=0
61742025-04-30T06:42:56.876ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "295dc350-6241-46fb-9fb7-8892d6684c2c", "content-length": "109", "date": "Wed, 30 Apr 2025 06:42:56 GMT"} })
61752025-04-30T06:42:56.876ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/1/component/sp/caboose?firmware_slot=1
61762025-04-30T06:42:56.878ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "4968e367-1be8-4fff-bd26-1755d9737956", "content-length": "109", "date": "Wed, 30 Apr 2025 06:42:56 GMT"} })
61772025-04-30T06:42:56.878ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/1/component/rot/caboose?firmware_slot=0
61782025-04-30T06:42:56.880ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "6f621cd0-8f91-4d2e-8723-d383fd96cfe3", "content-length": "169", "date": "Wed, 30 Apr 2025 06:42:56 GMT"} })
61792025-04-30T06:42:56.880ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/1/component/rot/caboose?firmware_slot=1
61802025-04-30T06:42:56.882ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "1cfa5988-67a9-4c08-81b7-83007432ae30", "content-length": "169", "date": "Wed, 30 Apr 2025 06:42:56 GMT"} })
61812025-04-30T06:42:56.882ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/1/component/stage0/caboose?firmware_slot=0
61822025-04-30T06:42:56.885ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "c14be751-8013-4357-968d-f463fe9ee319", "content-length": "178", "date": "Wed, 30 Apr 2025 06:42:56 GMT"} })
61832025-04-30T06:42:56.885ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/1/component/stage0/caboose?firmware_slot=1
61842025-04-30T06:42:56.887ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "546f74dc-d475-49a1-a653-30408fe0a1df", "content-length": "178", "date": "Wed, 30 Apr 2025 06:42:56 GMT"} })
61852025-04-30T06:42:56.887ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/1/component/rot/cmpa
61862025-04-30T06:42:56.888ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "8ff74aa3-dabc-4a34-98fc-8ccc0adaace7", "content-length": "702", "date": "Wed, 30 Apr 2025 06:42:56 GMT"} })
61872025-04-30T06:42:56.888ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/1/component/rot/cfpa
61882025-04-30T06:42:56.889ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "b3caf62c-0e63-4a61-8a10-287921a755e8", "content-length": "727", "date": "Wed, 30 Apr 2025 06:42:56 GMT"} })
61892025-04-30T06:42:56.889ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/1/component/rot/cfpa
61902025-04-30T06:42:56.890ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "e8e6817d-4fcf-4d91-8774-07d2b69e1bb1", "content-length": "729", "date": "Wed, 30 Apr 2025 06:42:56 GMT"} })
61912025-04-30T06:42:56.890ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/1/component/rot/cfpa
61922025-04-30T06:42:56.891ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "d428788e-8fba-40e9-b6a6-0e289bd7118b", "content-length": "728", "date": "Wed, 30 Apr 2025 06:42:56 GMT"} })
61932025-04-30T06:42:56.891ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
61942025-04-30T06:42:56.891ZTRACtest_snapshot_basic (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
61952025-04-30T06:42:56.891ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
61962025-04-30T06:42:56.891ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
61972025-04-30T06:42:56.892ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
61982025-04-30T06:42:56.895ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
61992025-04-30T06:42:56.899ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
62002025-04-30T06:42:56.899ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
62012025-04-30T06:42:56.899ZTRACtest_snapshot_basic (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
62022025-04-30T06:42:56.899ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
62032025-04-30T06:42:56.899ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
62042025-04-30T06:42:56.900ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
62052025-04-30T06:42:56.900ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
62062025-04-30T06:42:56.900ZTRACtest_snapshot_basic (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
62072025-04-30T06:42:56.900ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
62082025-04-30T06:42:56.901ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {} }
62092025-04-30T06:42:56.901ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
62102025-04-30T06:42:56.903ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
62112025-04-30T06:42:56.907ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
62122025-04-30T06:42:56.907ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
62132025-04-30T06:42:56.907ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
62142025-04-30T06:42:56.907ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
62152025-04-30T06:42:56.912ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
62162025-04-30T06:42:56.920ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
62172025-04-30T06:42:56.920ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
62182025-04-30T06:42:56.920ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {} }
62192025-04-30T06:42:56.921ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
62202025-04-30T06:42:56.992ZDEBGtest_snapshot_basic (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:41923
62212025-04-30T06:42:56.992ZDEBGtest_snapshot_basic (BackgroundTasks): client request SledAgent = http://[::1]:41923 background_task = inventory_collection body = None method = GET uri = http://[::1]:41923/inventory
62222025-04-30T06:42:56.994ZINFOtest_snapshot_basic (dropshot): accepted connection local_addr = [::1]:41923 remote_addr = [::1]:43848 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
62232025-04-30T06:42:56.994ZTRACtest_snapshot_basic (dropshot): incoming request local_addr = [::1]:41923 method = GET remote_addr = [::1]:43848 req_id = f07b4e6b-c057-4e7b-a4b4-b8af4a71c375 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
62242025-04-30T06:42:56.995ZINFOtest_snapshot_basic (dropshot): request completed latency_us = 706 local_addr = [::1]:41923 method = GET remote_addr = [::1]:43848 req_id = f07b4e6b-c057-4e7b-a4b4-b8af4a71c375 response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
62252025-04-30T06:42:56.995ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
62262025-04-30T06:42:56.996ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {} }
62272025-04-30T06:42:56.996ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
62282025-04-30T06:42:56.997ZDEBGtest_snapshot_basic (BackgroundTasks): client response SledAgent = http://[::1]:41923 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:41923/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "f07b4e6b-c057-4e7b-a4b4-b8af4a71c375", "content-length": "3141", "date": "Wed, 30 Apr 2025 06:42:56 GMT"} })
62292025-04-30T06:42:56.998ZDEBGtest_snapshot_basic (BackgroundTasks): begin collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
62302025-04-30T06:42:56.999ZDEBGtest_snapshot_basic (BackgroundTasks): end collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
62312025-04-30T06:42:56.999ZDEBGtest_snapshot_basic (BackgroundTasks): finished collection background_task = inventory_collection
62322025-04-30T06:42:57.001ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
62332025-04-30T06:42:57.001ZTRACtest_snapshot_basic (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
62342025-04-30T06:42:57.001ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
62352025-04-30T06:42:57.001ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
62362025-04-30T06:42:57.002ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
62372025-04-30T06:42:57.008ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
62382025-04-30T06:42:57.011ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
62392025-04-30T06:42:57.012ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
62402025-04-30T06:42:57.012ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
62412025-04-30T06:42:57.012ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
62422025-04-30T06:42:57.040ZDEBGtest_snapshot_basic: attempt to fetch latest inventory collection result = Ok("found")
62432025-04-30T06:42:57.040ZDEBGtest_snapshot_basic (results-sink): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
62442025-04-30T06:42:57.040ZDEBGtest_snapshot_basic (results-sink-cluster): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
62452025-04-30T06:42:57.041ZWARNtest_snapshot_basic (datastore): Non-retryable transaction failure err = DatabaseError(UniqueViolation, "duplicate key value violates unique constraint \\"inv_caboose_pkey\\"") transaction = inventory_insert_collection
62462025-04-30T06:42:57.041ZWARNtest_snapshot_basic (BackgroundTasks): inventory collection failed background_task = inventory_collection error = failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint "inv_caboose_pkey"
62472025-04-30T06:42:57.041ZDEBGtest_snapshot_basic (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 407.891858ms iteration = 17 status = {"error":"failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint \\"inv_caboose_pkey\\""}
62482025-04-30T06:42:57.041ZDEBGtest_snapshot_basic (BackgroundTasks): activating background_task = inventory_collection iteration = 18 reason = Signaled
62492025-04-30T06:42:57.041ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
62502025-04-30T06:42:57.042ZTRACtest_snapshot_basic (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
62512025-04-30T06:42:57.042ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
62522025-04-30T06:42:57.042ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
62532025-04-30T06:42:57.042ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
62542025-04-30T06:42:57.043ZDEBGtest_snapshot_basic (clickhouse-client): unrolling 1 total samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 6ad06c6b-9a6e-4ed1-b3f4-5d2e9da3f8a7
62552025-04-30T06:42:57.043ZTRACtest_snapshot_basic (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 6ad06c6b-9a6e-4ed1-b3f4-5d2e9da3f8a7 n_columns = 4 n_rows = 2 sql = INSERT INTO oximeter.fields_string FORMAT Native
62562025-04-30T06:42:57.049ZDEBGtest_snapshot_basic (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 6ad06c6b-9a6e-4ed1-b3f4-5d2e9da3f8a7 n_rows = 2 table_name = fields_string
62572025-04-30T06:42:57.050ZTRACtest_snapshot_basic (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 6ad06c6b-9a6e-4ed1-b3f4-5d2e9da3f8a7 n_columns = 4 n_rows = 1 sql = INSERT INTO oximeter.measurements_i64 FORMAT Native
62582025-04-30T06:42:57.050ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
62592025-04-30T06:42:57.054ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
62602025-04-30T06:42:57.059ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
62612025-04-30T06:42:57.059ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
62622025-04-30T06:42:57.060ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
62632025-04-30T06:42:57.064ZDEBGtest_snapshot_basic (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 6ad06c6b-9a6e-4ed1-b3f4-5d2e9da3f8a7 n_rows = 1 table_name = measurements_i64
62642025-04-30T06:42:57.064ZTRACtest_snapshot_basic (results-sink): successfully inserted samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
62652025-04-30T06:42:57.064ZDEBGtest_snapshot_basic (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = [(d1ad44e5-18dd-4193-a70e-bf9042f0e0e1, 0)]
62662025-04-30T06:42:57.065ZTRACtest_snapshot_basic (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
62672025-04-30T06:42:57.065ZTRACtest_snapshot_basic (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: f2 d8 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 ............._mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:58462 req_id = b928db31-5dcd-4ee7-9bc1-7c1e2bf11f73
62682025-04-30T06:42:57.065ZDEBGtest_snapshot_basic (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 62168,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:58462 req_id = b928db31-5dcd-4ee7-9bc1-7c1e2bf11f73
62692025-04-30T06:42:57.066ZDEBGtest_snapshot_basic (store): query key key = _mgs._tcp
62702025-04-30T06:42:57.066ZDEBGtest_snapshot_basic (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
62712025-04-30T06:42:57.066ZDEBGtest_snapshot_basic (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:58462 query = LowerQuery { name: LowerName(Name("_mgs._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 41502, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = b928db31-5dcd-4ee7-9bc1-7c1e2bf11f73
62722025-04-30T06:42:57.066ZDEBGtest_snapshot_basic (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 41502, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 4241, tv_nsec: 865507980 } })
62732025-04-30T06:42:57.066ZTRACtest_snapshot_basic (DnsResolver): lookup_service_targets: looking up SRV target
62742025-04-30T06:42:57.066ZTRACtest_snapshot_basic (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: 65 f9 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e e...........-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:58929 req_id = d67497d3-0ee6-4e46-bec2-e7995659b6b8
62752025-04-30T06:42:57.067ZDEBGtest_snapshot_basic (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 26105,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:58929 req_id = d67497d3-0ee6-4e46-bec2-e7995659b6b8
62762025-04-30T06:42:57.067ZDEBGtest_snapshot_basic (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
62772025-04-30T06:42:57.067ZDEBGtest_snapshot_basic (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:58929 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = d67497d3-0ee6-4e46-bec2-e7995659b6b8
62782025-04-30T06:42:57.134ZTRACtest_snapshot_basic (DnsResolver): lookup_all_socket_v6 srv dns_name = _clickhouse-admin-keeper._tcp.control-plane.oxide.internal
62792025-04-30T06:42:57.134ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
62802025-04-30T06:42:57.135ZTRACtest_snapshot_basic (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
62812025-04-30T06:42:57.135ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
62822025-04-30T06:42:57.135ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {} }
62832025-04-30T06:42:57.137ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
62842025-04-30T06:42:57.137ZTRACtest_snapshot_basic (dns): buffer buffer = Length: 87 (0x57) bytes\n0000: 93 ca 01 00 00 01 00 00 00 00 00 01 18 5f 63 6c ............._cl\n0010: 69 63 6b 68 6f 75 73 65 2d 61 64 6d 69 6e 2d 6b ickhouse-admin-k\n0020: 65 65 70 65 72 04 5f 74 63 70 0d 63 6f 6e 74 72 eeper._tcp.contr\n0030: 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 69 ol-plane.oxide.i\n0040: 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 04 nternal..!....).\n0050: d0 00 00 00 00 00 00 ....... kind = dns peer_addr = [::1]:60659 req_id = 0aa1d54f-18d7-4a54-80b2-382fcf037723
62852025-04-30T06:42:57.137ZDEBGtest_snapshot_basic (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 37834,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 24,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 97,\n 100,\n 109,\n 105,\n 110,\n 45,\n 107,\n 101,\n 101,\n 112,\n 101,\n 114,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:60659 req_id = 0aa1d54f-18d7-4a54-80b2-382fcf037723
62862025-04-30T06:42:57.137ZDEBGtest_snapshot_basic (store): query key key = _clickhouse-admin-keeper._tcp
62872025-04-30T06:42:57.137ZERROtest_snapshot_basic (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 37834, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } NXDOMAIN: no records found for name: "_clickhouse-admin-keeper._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:60659 req_id = 0aa1d54f-18d7-4a54-80b2-382fcf037723
62882025-04-30T06:42:57.137ZDEBGtest_snapshot_basic (BackgroundTasks): begin collection background_task = inventory_collection
62892025-04-30T06:42:57.137ZDEBGtest_snapshot_basic (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:41502
62902025-04-30T06:42:57.137ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/ignition
62912025-04-30T06:42:57.138ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
62922025-04-30T06:42:57.145ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
62932025-04-30T06:42:57.145ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
62942025-04-30T06:42:57.145ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {} }
62952025-04-30T06:42:57.145ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
62962025-04-30T06:42:57.147ZWARNtest_snapshot_basic (results-sink-cluster): failed to insert some results into metric DB collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 error = Failed to check out connection to database: No backends found for this service file = oximeter/collector/src/results_sink.rs:92
62972025-04-30T06:42:57.148ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "e269a147-1941-49df-883c-2e3627ab5820", "content-length": "835", "date": "Wed, 30 Apr 2025 06:42:56 GMT"} })
62982025-04-30T06:42:57.148ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/0
62992025-04-30T06:42:57.148ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
63002025-04-30T06:42:57.148ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {} }
63012025-04-30T06:42:57.149ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
63022025-04-30T06:42:57.151ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "8a94f822-8da4-408f-b0ac-f5efe5d4a4df", "content-length": "734", "date": "Wed, 30 Apr 2025 06:42:56 GMT"} })
63032025-04-30T06:42:57.151ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/0/component/sp/caboose?firmware_slot=0
63042025-04-30T06:42:57.157ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "07f72258-5786-46b4-8ad9-7407a7b5a89a", "content-length": "111", "date": "Wed, 30 Apr 2025 06:42:56 GMT"} })
63052025-04-30T06:42:57.157ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/0/component/sp/caboose?firmware_slot=1
63062025-04-30T06:42:57.164ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "3c5ffb6a-d83d-45c0-8aab-a58802fbb852", "content-length": "111", "date": "Wed, 30 Apr 2025 06:42:56 GMT"} })
63072025-04-30T06:42:57.164ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/0/component/rot/caboose?firmware_slot=0
63082025-04-30T06:42:57.170ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "cf855306-fab4-4a38-a716-05dadc673a88", "content-length": "170", "date": "Wed, 30 Apr 2025 06:42:56 GMT"} })
63092025-04-30T06:42:57.170ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/0/component/rot/caboose?firmware_slot=1
63102025-04-30T06:42:57.174ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "6033f428-9ebc-43eb-95dc-3c97673d33a1", "content-length": "170", "date": "Wed, 30 Apr 2025 06:42:56 GMT"} })
63112025-04-30T06:42:57.174ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/0/component/stage0/caboose?firmware_slot=0
63122025-04-30T06:42:57.179ZDEBGtest_snapshot_basic: attempt to fetch latest inventory collection result = Ok("found")
63132025-04-30T06:42:57.181ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "8a16f455-84d1-4a37-be9b-629bfde5bff5", "content-length": "179", "date": "Wed, 30 Apr 2025 06:42:56 GMT"} })
63142025-04-30T06:42:57.182ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/0/component/stage0/caboose?firmware_slot=1
63152025-04-30T06:42:57.186ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "ae4d95c5-b870-4be3-ad6a-3d2870faefec", "content-length": "179", "date": "Wed, 30 Apr 2025 06:42:56 GMT"} })
63162025-04-30T06:42:57.186ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/0/component/rot/cmpa
63172025-04-30T06:42:57.189ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "183e8a07-277b-4f31-bf3f-9d85ba115741", "content-length": "702", "date": "Wed, 30 Apr 2025 06:42:56 GMT"} })
63182025-04-30T06:42:57.189ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/0/component/rot/cfpa
63192025-04-30T06:42:57.196ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "30375fd0-4a03-49cf-80a0-91cf9eeb5059", "content-length": "727", "date": "Wed, 30 Apr 2025 06:42:56 GMT"} })
63202025-04-30T06:42:57.196ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/0/component/rot/cfpa
63212025-04-30T06:42:57.198ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "ceb2625e-bc3d-44f8-867b-4889e75d290e", "content-length": "729", "date": "Wed, 30 Apr 2025 06:42:56 GMT"} })
63222025-04-30T06:42:57.198ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/0/component/rot/cfpa
63232025-04-30T06:42:57.199ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "4741ac11-bdfa-4bc8-9bdc-ca5280e5c0c2", "content-length": "728", "date": "Wed, 30 Apr 2025 06:42:56 GMT"} })
63242025-04-30T06:42:57.199ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/1
63252025-04-30T06:42:57.201ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "a1f631e5-8356-4099-9f8b-45a209b48e99", "content-length": "734", "date": "Wed, 30 Apr 2025 06:42:56 GMT"} })
63262025-04-30T06:42:57.201ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/1/component/sp/caboose?firmware_slot=0
63272025-04-30T06:42:57.205ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "805c218b-03f6-43f5-bb82-145770a6a56b", "content-length": "111", "date": "Wed, 30 Apr 2025 06:42:56 GMT"} })
63282025-04-30T06:42:57.205ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/1/component/sp/caboose?firmware_slot=1
63292025-04-30T06:42:57.209ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "41b85f9d-eb19-42c3-be03-eddf70ee133c", "content-length": "111", "date": "Wed, 30 Apr 2025 06:42:56 GMT"} })
63302025-04-30T06:42:57.210ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/1/component/rot/caboose?firmware_slot=0
63312025-04-30T06:42:57.211ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "13846940-b2fa-4aaa-8fff-8c2fc61bb7e1", "content-length": "170", "date": "Wed, 30 Apr 2025 06:42:56 GMT"} })
63322025-04-30T06:42:57.211ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/1/component/rot/caboose?firmware_slot=1
63332025-04-30T06:42:57.213ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "01dfd996-fc2f-4b95-9d0e-cbc27091b92a", "content-length": "170", "date": "Wed, 30 Apr 2025 06:42:56 GMT"} })
63342025-04-30T06:42:57.213ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/1/component/stage0/caboose?firmware_slot=0
63352025-04-30T06:42:57.214ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "d48d5fe8-45ef-4db5-b4a9-81dd41042f67", "content-length": "179", "date": "Wed, 30 Apr 2025 06:42:56 GMT"} })
63362025-04-30T06:42:57.214ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/1/component/stage0/caboose?firmware_slot=1
63372025-04-30T06:42:57.216ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "b05f0ba0-3d30-4e97-8e3c-41868deb78a3", "content-length": "179", "date": "Wed, 30 Apr 2025 06:42:56 GMT"} })
63382025-04-30T06:42:57.216ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/1/component/rot/cmpa
63392025-04-30T06:42:57.218ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "71874627-1768-40a0-a742-372490984859", "content-length": "702", "date": "Wed, 30 Apr 2025 06:42:56 GMT"} })
63402025-04-30T06:42:57.218ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/1/component/rot/cfpa
63412025-04-30T06:42:57.219ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "2a2958e0-a7e3-4ab5-b4b8-dc0426868975", "content-length": "727", "date": "Wed, 30 Apr 2025 06:42:56 GMT"} })
63422025-04-30T06:42:57.219ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/1/component/rot/cfpa
63432025-04-30T06:42:57.220ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "859800f9-8c29-4112-ba7d-65cce0d49c6d", "content-length": "729", "date": "Wed, 30 Apr 2025 06:42:56 GMT"} })
63442025-04-30T06:42:57.220ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/1/component/rot/cfpa
63452025-04-30T06:42:57.223ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "6830ebc2-8049-4ed7-9e14-c42e93ed6eb3", "content-length": "728", "date": "Wed, 30 Apr 2025 06:42:56 GMT"} })
63462025-04-30T06:42:57.223ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/0
63472025-04-30T06:42:57.224ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "b409e332-9b3f-4b72-88dd-b41f9109db9f", "content-length": "723", "date": "Wed, 30 Apr 2025 06:42:56 GMT"} })
63482025-04-30T06:42:57.224ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/0/component/sp/caboose?firmware_slot=0
63492025-04-30T06:42:57.226ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "3caf2628-09d0-4629-9db9-bf6f87327245", "content-length": "109", "date": "Wed, 30 Apr 2025 06:42:56 GMT"} })
63502025-04-30T06:42:57.226ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/0/component/sp/caboose?firmware_slot=1
63512025-04-30T06:42:57.228ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "ce1ba472-84e2-4508-a1f7-aece06193265", "content-length": "109", "date": "Wed, 30 Apr 2025 06:42:56 GMT"} })
63522025-04-30T06:42:57.228ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/0/component/rot/caboose?firmware_slot=0
63532025-04-30T06:42:57.230ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
63542025-04-30T06:42:57.230ZTRACtest_snapshot_basic (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
63552025-04-30T06:42:57.230ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
63562025-04-30T06:42:57.230ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {} }
63572025-04-30T06:42:57.231ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
63582025-04-30T06:42:57.234ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "dad8b295-ad03-4e10-9b25-9048149f5c6a", "content-length": "169", "date": "Wed, 30 Apr 2025 06:42:56 GMT"} })
63592025-04-30T06:42:57.234ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/0/component/rot/caboose?firmware_slot=1
63602025-04-30T06:42:57.234ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
63612025-04-30T06:42:57.241ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
63622025-04-30T06:42:57.241ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
63632025-04-30T06:42:57.241ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {} }
63642025-04-30T06:42:57.242ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
63652025-04-30T06:42:57.246ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
63662025-04-30T06:42:57.246ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {} }
63672025-04-30T06:42:57.252ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
63682025-04-30T06:42:57.254ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "22261889-e7fc-4f49-9e24-526cf9384ed8", "content-length": "169", "date": "Wed, 30 Apr 2025 06:42:56 GMT"} })
63692025-04-30T06:42:57.254ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/0/component/stage0/caboose?firmware_slot=0
63702025-04-30T06:42:57.259ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "9e45d57c-9b97-434f-bae4-22079f6b9400", "content-length": "178", "date": "Wed, 30 Apr 2025 06:42:56 GMT"} })
63712025-04-30T06:42:57.260ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/0/component/stage0/caboose?firmware_slot=1
63722025-04-30T06:42:57.265ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "c80eb699-95dc-46b1-bdbc-e103afc8889a", "content-length": "178", "date": "Wed, 30 Apr 2025 06:42:56 GMT"} })
63732025-04-30T06:42:57.265ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/0/component/rot/cmpa
63742025-04-30T06:42:57.267ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "76e7e9fa-bfbb-403a-b1e4-13fa5d48318b", "content-length": "702", "date": "Wed, 30 Apr 2025 06:42:56 GMT"} })
63752025-04-30T06:42:57.267ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/0/component/rot/cfpa
63762025-04-30T06:42:57.268ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "25004a24-9332-4461-adcb-ec32e10e7ce7", "content-length": "727", "date": "Wed, 30 Apr 2025 06:42:56 GMT"} })
63772025-04-30T06:42:57.268ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/0/component/rot/cfpa
63782025-04-30T06:42:57.269ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "f2686a24-6d27-4abb-9255-b83282e8a1c8", "content-length": "729", "date": "Wed, 30 Apr 2025 06:42:56 GMT"} })
63792025-04-30T06:42:57.269ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/0/component/rot/cfpa
63802025-04-30T06:42:57.271ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "cac76033-19bd-4c34-80ee-897bac95022e", "content-length": "728", "date": "Wed, 30 Apr 2025 06:42:56 GMT"} })
63812025-04-30T06:42:57.271ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/1
63822025-04-30T06:42:57.272ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "713b52af-09b3-4310-b27b-8a5a24e8a0c3", "content-length": "723", "date": "Wed, 30 Apr 2025 06:42:56 GMT"} })
63832025-04-30T06:42:57.273ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/1/component/sp/caboose?firmware_slot=0
63842025-04-30T06:42:57.275ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "cc5261b8-fa57-40ee-8d79-2ac281ac2bbf", "content-length": "109", "date": "Wed, 30 Apr 2025 06:42:56 GMT"} })
63852025-04-30T06:42:57.275ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/1/component/sp/caboose?firmware_slot=1
63862025-04-30T06:42:57.278ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "9a1f6aba-f19c-4cb1-baac-f7b0e0a986c7", "content-length": "109", "date": "Wed, 30 Apr 2025 06:42:56 GMT"} })
63872025-04-30T06:42:57.278ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/1/component/rot/caboose?firmware_slot=0
63882025-04-30T06:42:57.282ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "a0951774-02e3-4baa-bbd8-1c3cf33ab3d5", "content-length": "169", "date": "Wed, 30 Apr 2025 06:42:56 GMT"} })
63892025-04-30T06:42:57.282ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/1/component/rot/caboose?firmware_slot=1
63902025-04-30T06:42:57.283ZDEBGtest_snapshot_basic: attempt to fetch latest inventory collection result = Ok("found")
63912025-04-30T06:42:57.286ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "94c31266-5ffb-4559-9fc6-ea63aee5eb33", "content-length": "169", "date": "Wed, 30 Apr 2025 06:42:56 GMT"} })
63922025-04-30T06:42:57.286ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/1/component/stage0/caboose?firmware_slot=0
63932025-04-30T06:42:57.288ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "96e2d7c6-56b3-4217-8d65-597ac6956e34", "content-length": "178", "date": "Wed, 30 Apr 2025 06:42:56 GMT"} })
63942025-04-30T06:42:57.288ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/1/component/stage0/caboose?firmware_slot=1
63952025-04-30T06:42:57.293ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "2baeabbe-e438-4ca7-8d13-0781a58fdcfd", "content-length": "178", "date": "Wed, 30 Apr 2025 06:42:56 GMT"} })
63962025-04-30T06:42:57.294ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/1/component/rot/cmpa
63972025-04-30T06:42:57.296ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "44bf99f7-0385-4e34-b617-4b0d3664bbbc", "content-length": "702", "date": "Wed, 30 Apr 2025 06:42:56 GMT"} })
63982025-04-30T06:42:57.296ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/1/component/rot/cfpa
63992025-04-30T06:42:57.297ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "1b1dd02a-eef1-4f1d-8cb6-39f784f1dac5", "content-length": "727", "date": "Wed, 30 Apr 2025 06:42:56 GMT"} })
64002025-04-30T06:42:57.297ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/1/component/rot/cfpa
64012025-04-30T06:42:57.300ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "f9db3135-1806-4ff4-82c7-2aeee5e754c4", "content-length": "729", "date": "Wed, 30 Apr 2025 06:42:56 GMT"} })
64022025-04-30T06:42:57.300ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/1/component/rot/cfpa
64032025-04-30T06:42:57.303ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "dfa171f9-bbb0-4a84-b372-1fe095b2d6c0", "content-length": "728", "date": "Wed, 30 Apr 2025 06:42:56 GMT"} })
64042025-04-30T06:42:57.303ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
64052025-04-30T06:42:57.303ZTRACtest_snapshot_basic (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
64062025-04-30T06:42:57.303ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
64072025-04-30T06:42:57.303ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
64082025-04-30T06:42:57.306ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
64092025-04-30T06:42:57.314ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
64102025-04-30T06:42:57.321ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
64112025-04-30T06:42:57.321ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
64122025-04-30T06:42:57.321ZTRACtest_snapshot_basic (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
64132025-04-30T06:42:57.321ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
64142025-04-30T06:42:57.321ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
64152025-04-30T06:42:57.322ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
64162025-04-30T06:42:57.326ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
64172025-04-30T06:42:57.333ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
64182025-04-30T06:42:57.333ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
64192025-04-30T06:42:57.334ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
64202025-04-30T06:42:57.334ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
64212025-04-30T06:42:57.410ZDEBGtest_snapshot_basic (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:41923
64222025-04-30T06:42:57.410ZDEBGtest_snapshot_basic (BackgroundTasks): client request SledAgent = http://[::1]:41923 background_task = inventory_collection body = None method = GET uri = http://[::1]:41923/inventory
64232025-04-30T06:42:57.411ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
64242025-04-30T06:42:57.411ZTRACtest_snapshot_basic (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
64252025-04-30T06:42:57.411ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
64262025-04-30T06:42:57.411ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {} }
64272025-04-30T06:42:57.412ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
64282025-04-30T06:42:57.412ZINFOtest_snapshot_basic (dropshot): accepted connection local_addr = [::1]:41923 remote_addr = [::1]:55685 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
64292025-04-30T06:42:57.414ZTRACtest_snapshot_basic (dropshot): incoming request local_addr = [::1]:41923 method = GET remote_addr = [::1]:55685 req_id = 6d79bc9c-8520-493c-b6dc-ca39d90d5d3c sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
64302025-04-30T06:42:57.414ZINFOtest_snapshot_basic (dropshot): request completed latency_us = 371 local_addr = [::1]:41923 method = GET remote_addr = [::1]:55685 req_id = 6d79bc9c-8520-493c-b6dc-ca39d90d5d3c response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
64312025-04-30T06:42:57.416ZDEBGtest_snapshot_basic (BackgroundTasks): client response SledAgent = http://[::1]:41923 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:41923/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "6d79bc9c-8520-493c-b6dc-ca39d90d5d3c", "content-length": "3141", "date": "Wed, 30 Apr 2025 06:42:57 GMT"} })
64322025-04-30T06:42:57.419ZDEBGtest_snapshot_basic (BackgroundTasks): begin collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
64332025-04-30T06:42:57.419ZDEBGtest_snapshot_basic (BackgroundTasks): end collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
64342025-04-30T06:42:57.419ZDEBGtest_snapshot_basic (BackgroundTasks): finished collection background_task = inventory_collection
64352025-04-30T06:42:57.424ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
64362025-04-30T06:42:57.424ZTRACtest_snapshot_basic (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
64372025-04-30T06:42:57.424ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
64382025-04-30T06:42:57.424ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
64392025-04-30T06:42:57.425ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
64402025-04-30T06:42:57.425ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
64412025-04-30T06:42:57.431ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
64422025-04-30T06:42:57.431ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
64432025-04-30T06:42:57.431ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {} }
64442025-04-30T06:42:57.431ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
64452025-04-30T06:42:57.439ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
64462025-04-30T06:42:57.445ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
64472025-04-30T06:42:57.445ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
64482025-04-30T06:42:57.445ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
64492025-04-30T06:42:57.447ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
64502025-04-30T06:42:57.449ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
64512025-04-30T06:42:57.449ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {} }
64522025-04-30T06:42:57.449ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
64532025-04-30T06:42:57.480ZWARNtest_snapshot_basic (datastore): Non-retryable transaction failure err = DatabaseError(UniqueViolation, "duplicate key value violates unique constraint \\"inv_caboose_pkey\\"") transaction = inventory_insert_collection
64542025-04-30T06:42:57.481ZWARNtest_snapshot_basic (BackgroundTasks): inventory collection failed background_task = inventory_collection error = failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint "inv_caboose_pkey"
64552025-04-30T06:42:57.481ZDEBGtest_snapshot_basic (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 439.367812ms iteration = 18 status = {"error":"failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint \\"inv_caboose_pkey\\""}
64562025-04-30T06:42:57.481ZDEBGtest_snapshot_basic (BackgroundTasks): activating background_task = inventory_collection iteration = 19 reason = Signaled
64572025-04-30T06:42:57.481ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
64582025-04-30T06:42:57.481ZTRACtest_snapshot_basic (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
64592025-04-30T06:42:57.481ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
64602025-04-30T06:42:57.481ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
64612025-04-30T06:42:57.481ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
64622025-04-30T06:42:57.485ZDEBGtest_snapshot_basic: attempt to fetch latest inventory collection result = Ok("found")
64632025-04-30T06:42:57.485ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
64642025-04-30T06:42:57.490ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
64652025-04-30T06:42:57.491ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
64662025-04-30T06:42:57.491ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
64672025-04-30T06:42:57.491ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
64682025-04-30T06:42:57.495ZDEBGtest_snapshot_basic (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = [(d1ad44e5-18dd-4193-a70e-bf9042f0e0e1, 0)]
64692025-04-30T06:42:57.495ZTRACtest_snapshot_basic (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
64702025-04-30T06:42:57.497ZTRACtest_snapshot_basic (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: c9 e5 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 ............._mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:62202 req_id = dac8c815-31be-4b4a-a1b8-8934ee0e36d1
64712025-04-30T06:42:57.497ZDEBGtest_snapshot_basic (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 51685,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:62202 req_id = dac8c815-31be-4b4a-a1b8-8934ee0e36d1
64722025-04-30T06:42:57.497ZDEBGtest_snapshot_basic (store): query key key = _mgs._tcp
64732025-04-30T06:42:57.497ZDEBGtest_snapshot_basic (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
64742025-04-30T06:42:57.497ZDEBGtest_snapshot_basic (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:62202 query = LowerQuery { name: LowerName(Name("_mgs._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 41502, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = dac8c815-31be-4b4a-a1b8-8934ee0e36d1
64752025-04-30T06:42:57.497ZDEBGtest_snapshot_basic (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 41502, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 4242, tv_nsec: 296439263 } })
64762025-04-30T06:42:57.497ZTRACtest_snapshot_basic (DnsResolver): lookup_service_targets: looking up SRV target
64772025-04-30T06:42:57.498ZTRACtest_snapshot_basic (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: 63 57 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e cW..........-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:58381 req_id = e4c1e171-0738-4d1f-b582-733168ad5bd9
64782025-04-30T06:42:57.498ZDEBGtest_snapshot_basic (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 25431,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:58381 req_id = e4c1e171-0738-4d1f-b582-733168ad5bd9
64792025-04-30T06:42:57.498ZDEBGtest_snapshot_basic (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
64802025-04-30T06:42:57.499ZDEBGtest_snapshot_basic (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:58381 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = e4c1e171-0738-4d1f-b582-733168ad5bd9
64812025-04-30T06:42:57.545ZTRACtest_snapshot_basic (DnsResolver): lookup_all_socket_v6 srv dns_name = _clickhouse-admin-keeper._tcp.control-plane.oxide.internal
64822025-04-30T06:42:57.545ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
64832025-04-30T06:42:57.545ZTRACtest_snapshot_basic (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
64842025-04-30T06:42:57.545ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
64852025-04-30T06:42:57.546ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {} }
64862025-04-30T06:42:57.546ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
64872025-04-30T06:42:57.547ZTRACtest_snapshot_basic (dns): buffer buffer = Length: 87 (0x57) bytes\n0000: d8 11 01 00 00 01 00 00 00 00 00 01 18 5f 63 6c ............._cl\n0010: 69 63 6b 68 6f 75 73 65 2d 61 64 6d 69 6e 2d 6b ickhouse-admin-k\n0020: 65 65 70 65 72 04 5f 74 63 70 0d 63 6f 6e 74 72 eeper._tcp.contr\n0030: 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 69 ol-plane.oxide.i\n0040: 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 04 nternal..!....).\n0050: d0 00 00 00 00 00 00 ....... kind = dns peer_addr = [::1]:49646 req_id = 9b8926f0-b250-4afe-92ae-fbbbb3604321
64882025-04-30T06:42:57.548ZDEBGtest_snapshot_basic (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 55313,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 24,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 97,\n 100,\n 109,\n 105,\n 110,\n 45,\n 107,\n 101,\n 101,\n 112,\n 101,\n 114,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:49646 req_id = 9b8926f0-b250-4afe-92ae-fbbbb3604321
64892025-04-30T06:42:57.548ZDEBGtest_snapshot_basic (store): query key key = _clickhouse-admin-keeper._tcp
64902025-04-30T06:42:57.548ZERROtest_snapshot_basic (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 55313, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } NXDOMAIN: no records found for name: "_clickhouse-admin-keeper._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:49646 req_id = 9b8926f0-b250-4afe-92ae-fbbbb3604321
64912025-04-30T06:42:57.553ZDEBGtest_snapshot_basic (BackgroundTasks): begin collection background_task = inventory_collection
64922025-04-30T06:42:57.553ZDEBGtest_snapshot_basic (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:41502
64932025-04-30T06:42:57.553ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/ignition
64942025-04-30T06:42:57.555ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
64952025-04-30T06:42:57.561ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
64962025-04-30T06:42:57.561ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
64972025-04-30T06:42:57.561ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {} }
64982025-04-30T06:42:57.562ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
64992025-04-30T06:42:57.567ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
65002025-04-30T06:42:57.568ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {} }
65012025-04-30T06:42:57.568ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
65022025-04-30T06:42:57.591ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "419183ed-e019-4d51-9c9d-6b4cf3362302", "content-length": "835", "date": "Wed, 30 Apr 2025 06:42:57 GMT"} })
65032025-04-30T06:42:57.591ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/0
65042025-04-30T06:42:57.608ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "1551d1ab-3158-4ed7-83fb-8fa756b8ddab", "content-length": "734", "date": "Wed, 30 Apr 2025 06:42:57 GMT"} })
65052025-04-30T06:42:57.608ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/0/component/sp/caboose?firmware_slot=0
65062025-04-30T06:42:57.622ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "e5e9293a-0df9-4a0a-9f4b-71eac3ea9372", "content-length": "111", "date": "Wed, 30 Apr 2025 06:42:57 GMT"} })
65072025-04-30T06:42:57.628ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/0/component/sp/caboose?firmware_slot=1
65082025-04-30T06:42:57.633ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "bebe12ef-8dd4-4538-8bb2-7c8118042495", "content-length": "111", "date": "Wed, 30 Apr 2025 06:42:57 GMT"} })
65092025-04-30T06:42:57.633ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/0/component/rot/caboose?firmware_slot=0
65102025-04-30T06:42:57.638ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "00fd3769-82de-4f36-8c41-a50afe54c0fc", "content-length": "170", "date": "Wed, 30 Apr 2025 06:42:57 GMT"} })
65112025-04-30T06:42:57.638ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/0/component/rot/caboose?firmware_slot=1
65122025-04-30T06:42:57.646ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "746d4b68-b86f-4bd8-b1d5-684421324ddb", "content-length": "170", "date": "Wed, 30 Apr 2025 06:42:57 GMT"} })
65132025-04-30T06:42:57.646ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/0/component/stage0/caboose?firmware_slot=0
65142025-04-30T06:42:57.646ZDEBGtest_snapshot_basic: attempt to fetch latest inventory collection result = Ok("found")
65152025-04-30T06:42:57.652ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "d3b09e35-8555-4c36-a5a7-89f905c93beb", "content-length": "179", "date": "Wed, 30 Apr 2025 06:42:57 GMT"} })
65162025-04-30T06:42:57.652ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/0/component/stage0/caboose?firmware_slot=1
65172025-04-30T06:42:57.655ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "4a2dc6e2-b4ec-4c81-b452-6a7964fabbd7", "content-length": "179", "date": "Wed, 30 Apr 2025 06:42:57 GMT"} })
65182025-04-30T06:42:57.656ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/0/component/rot/cmpa
65192025-04-30T06:42:57.659ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "a18d9cf9-03f7-4cc4-8dba-1231868d96d4", "content-length": "702", "date": "Wed, 30 Apr 2025 06:42:57 GMT"} })
65202025-04-30T06:42:57.660ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/0/component/rot/cfpa
65212025-04-30T06:42:57.661ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "1a380e34-d832-4169-a5a4-928c483e55bc", "content-length": "727", "date": "Wed, 30 Apr 2025 06:42:57 GMT"} })
65222025-04-30T06:42:57.661ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/0/component/rot/cfpa
65232025-04-30T06:42:57.663ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "7e31decf-592a-45f0-990d-95cd24440df9", "content-length": "729", "date": "Wed, 30 Apr 2025 06:42:57 GMT"} })
65242025-04-30T06:42:57.663ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/0/component/rot/cfpa
65252025-04-30T06:42:57.669ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "7203f41e-1c34-4ba6-ace6-0de56d0d0ab3", "content-length": "728", "date": "Wed, 30 Apr 2025 06:42:57 GMT"} })
65262025-04-30T06:42:57.669ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/1
65272025-04-30T06:42:57.671ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "a498e812-86c1-4ef6-a06a-02fb8e5e605e", "content-length": "734", "date": "Wed, 30 Apr 2025 06:42:57 GMT"} })
65282025-04-30T06:42:57.671ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/1/component/sp/caboose?firmware_slot=0
65292025-04-30T06:42:57.675ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "ff23808d-1503-45a4-8f91-b9923c502491", "content-length": "111", "date": "Wed, 30 Apr 2025 06:42:57 GMT"} })
65302025-04-30T06:42:57.675ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/1/component/sp/caboose?firmware_slot=1
65312025-04-30T06:42:57.679ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "638dbbd2-2a5a-475a-bb2f-eca6fec0de69", "content-length": "111", "date": "Wed, 30 Apr 2025 06:42:57 GMT"} })
65322025-04-30T06:42:57.679ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/1/component/rot/caboose?firmware_slot=0
65332025-04-30T06:42:57.681ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "c8bc39d1-eb5c-4286-a6da-0a72c30af651", "content-length": "170", "date": "Wed, 30 Apr 2025 06:42:57 GMT"} })
65342025-04-30T06:42:57.681ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/1/component/rot/caboose?firmware_slot=1
65352025-04-30T06:42:57.704ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
65362025-04-30T06:42:57.704ZTRACtest_snapshot_basic (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
65372025-04-30T06:42:57.704ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
65382025-04-30T06:42:57.705ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {} }
65392025-04-30T06:42:57.706ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
65402025-04-30T06:42:57.710ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
65412025-04-30T06:42:57.718ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
65422025-04-30T06:42:57.718ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
65432025-04-30T06:42:57.718ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {} }
65442025-04-30T06:42:57.719ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
65452025-04-30T06:42:57.719ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "f7499449-a726-4ae6-8378-7f83f639bf8d", "content-length": "170", "date": "Wed, 30 Apr 2025 06:42:57 GMT"} })
65462025-04-30T06:42:57.720ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/1/component/stage0/caboose?firmware_slot=0
65472025-04-30T06:42:57.721ZINFOtest_snapshot_basic (oximeter-agent): refreshing list of producers from Nexus collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 file = oximeter/collector/src/agent.rs:562
65482025-04-30T06:42:57.722ZDEBGtest_snapshot_basic (oximeter): client request body = None method = GET uri = http://127.0.0.1:61468/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
65492025-04-30T06:42:57.723ZTRACtest_snapshot_basic (dropshot_internal): incoming request local_addr = 127.0.0.1:61468 method = GET remote_addr = 127.0.0.1:64332 req_id = 12a12208-8cc7-492c-9370-ec04805f5373 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
65502025-04-30T06:42:57.724ZTRACtest_snapshot_basic (dropshot_internal): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:61468 method = GET remote_addr = 127.0.0.1:64332 req_id = 12a12208-8cc7-492c-9370-ec04805f5373 resource = Database uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
65512025-04-30T06:42:57.724ZDEBGtest_snapshot_basic (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:61468 method = GET remote_addr = 127.0.0.1:64332 req_id = 12a12208-8cc7-492c-9370-ec04805f5373 roles = RoleSet { roles: {} } uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
65522025-04-30T06:42:57.724ZDEBGtest_snapshot_basic (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:61468 method = GET remote_addr = 127.0.0.1:64332 req_id = 12a12208-8cc7-492c-9370-ec04805f5373 resource = Database result = Ok(()) uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
65532025-04-30T06:42:57.725ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
65542025-04-30T06:42:57.725ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {} }
65552025-04-30T06:42:57.726ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
65562025-04-30T06:42:57.733ZINFOtest_snapshot_basic (dropshot_internal): request completed latency_us = 9904 local_addr = 127.0.0.1:61468 method = GET remote_addr = 127.0.0.1:64332 req_id = 12a12208-8cc7-492c-9370-ec04805f5373 response_code = 200 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
65572025-04-30T06:42:57.735ZDEBGtest_snapshot_basic (oximeter): client response result = Ok(Response { url: "http://127.0.0.1:61468/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending", status: 200, headers: {"content-type": "application/json", "x-request-id": "12a12208-8cc7-492c-9370-ec04805f5373", "content-length": "405", "date": "Wed, 30 Apr 2025 06:42:57 GMT"} })
65582025-04-30T06:42:57.736ZDEBGtest_snapshot_basic (oximeter): client request body = None method = GET uri = http://127.0.0.1:61468/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
65592025-04-30T06:42:57.738ZTRACtest_snapshot_basic (dropshot_internal): incoming request local_addr = 127.0.0.1:61468 method = GET remote_addr = 127.0.0.1:64332 req_id = 4fbd8095-f92f-4e07-ae56-6d28587a6f70 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
65602025-04-30T06:42:57.738ZTRACtest_snapshot_basic (dropshot_internal): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:61468 method = GET remote_addr = 127.0.0.1:64332 req_id = 4fbd8095-f92f-4e07-ae56-6d28587a6f70 resource = Database uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
65612025-04-30T06:42:57.739ZDEBGtest_snapshot_basic (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:61468 method = GET remote_addr = 127.0.0.1:64332 req_id = 4fbd8095-f92f-4e07-ae56-6d28587a6f70 roles = RoleSet { roles: {} } uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
65622025-04-30T06:42:57.739ZDEBGtest_snapshot_basic (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:61468 method = GET remote_addr = 127.0.0.1:64332 req_id = 4fbd8095-f92f-4e07-ae56-6d28587a6f70 resource = Database result = Ok(()) uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
65632025-04-30T06:42:57.743ZINFOtest_snapshot_basic (dropshot_internal): request completed latency_us = 4426 local_addr = 127.0.0.1:61468 method = GET remote_addr = 127.0.0.1:64332 req_id = 4fbd8095-f92f-4e07-ae56-6d28587a6f70 response_code = 200 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
65642025-04-30T06:42:57.744ZDEBGtest_snapshot_basic (oximeter): client response result = Ok(Response { url: "http://127.0.0.1:61468/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D", status: 200, headers: {"content-type": "application/json", "x-request-id": "4fbd8095-f92f-4e07-ae56-6d28587a6f70", "content-length": "29", "date": "Wed, 30 Apr 2025 06:42:57 GMT"} })
65652025-04-30T06:42:57.744ZTRACtest_snapshot_basic (oximeter-agent): ignoring request to update existing metric producer, since the endpoint information is the same as the existing collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = 354a52d4-3078-4706-81ce-12c32f6bbeaa
65662025-04-30T06:42:57.744ZTRACtest_snapshot_basic (oximeter-agent): ignoring request to update existing metric producer, since the endpoint information is the same as the existing collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
65672025-04-30T06:42:57.744ZINFOtest_snapshot_basic (oximeter-agent): refreshed list of producers from Nexus collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 file = oximeter/collector/src/agent.rs:637 n_current_tasks = 2 n_pruned_tasks = 0
65682025-04-30T06:42:57.744ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "040191ca-100d-408f-9c09-54829d180205", "content-length": "179", "date": "Wed, 30 Apr 2025 06:42:57 GMT"} })
65692025-04-30T06:42:57.744ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/1/component/stage0/caboose?firmware_slot=1
65702025-04-30T06:42:57.757ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "5b325475-247d-44ae-b01f-402f4a4f0b2e", "content-length": "179", "date": "Wed, 30 Apr 2025 06:42:57 GMT"} })
65712025-04-30T06:42:57.758ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/1/component/rot/cmpa
65722025-04-30T06:42:57.762ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "2ddb8983-c437-4138-a3cd-67f259c4a5ff", "content-length": "702", "date": "Wed, 30 Apr 2025 06:42:57 GMT"} })
65732025-04-30T06:42:57.762ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/1/component/rot/cfpa
65742025-04-30T06:42:57.763ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "e7af7791-7eea-42cd-b998-d5a4d8620992", "content-length": "727", "date": "Wed, 30 Apr 2025 06:42:57 GMT"} })
65752025-04-30T06:42:57.764ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/1/component/rot/cfpa
65762025-04-30T06:42:57.765ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "070b775d-3018-49eb-b572-327f4cfc9f3b", "content-length": "729", "date": "Wed, 30 Apr 2025 06:42:57 GMT"} })
65772025-04-30T06:42:57.769ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/1/component/rot/cfpa
65782025-04-30T06:42:57.784ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "b36d9b65-78b9-4666-9e68-78a21c65b48d", "content-length": "728", "date": "Wed, 30 Apr 2025 06:42:57 GMT"} })
65792025-04-30T06:42:57.784ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/0
65802025-04-30T06:42:57.785ZDEBGtest_snapshot_basic: attempt to fetch latest inventory collection result = Ok("found")
65812025-04-30T06:42:57.789ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "f2d5dbfb-c407-439c-bf20-a9ab6fb57537", "content-length": "723", "date": "Wed, 30 Apr 2025 06:42:57 GMT"} })
65822025-04-30T06:42:57.790ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/0/component/sp/caboose?firmware_slot=0
65832025-04-30T06:42:57.793ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "23384b44-4953-4c88-ac12-e7b5f72a03d3", "content-length": "109", "date": "Wed, 30 Apr 2025 06:42:57 GMT"} })
65842025-04-30T06:42:57.794ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/0/component/sp/caboose?firmware_slot=1
65852025-04-30T06:42:57.798ZDEBGtest_snapshot_basic (collection-task): sent timer-based collection request to the collection task collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
65862025-04-30T06:42:57.798ZDEBGtest_snapshot_basic (collection-task): collecting from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
65872025-04-30T06:42:57.800ZTRACtest_snapshot_basic (collection-task): sent collection request to producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
65882025-04-30T06:42:57.801ZDEBGtest_snapshot_basic (collection-task): collected results from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 n_results = 1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
65892025-04-30T06:42:57.801ZTRACtest_snapshot_basic (collection-task): forwarded results to main collection loop collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
65902025-04-30T06:42:57.801ZTRACtest_snapshot_basic (collection-task): top of inner collection loop, waiting for next request collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
65912025-04-30T06:42:57.803ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "e49c06b5-4237-43c7-8fa3-a99fd8aaf927", "content-length": "109", "date": "Wed, 30 Apr 2025 06:42:57 GMT"} })
65922025-04-30T06:42:57.803ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/0/component/rot/caboose?firmware_slot=0
65932025-04-30T06:42:57.807ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "6812ed6c-6bc1-4a52-a3dd-c576b955276a", "content-length": "169", "date": "Wed, 30 Apr 2025 06:42:57 GMT"} })
65942025-04-30T06:42:57.807ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/0/component/rot/caboose?firmware_slot=1
65952025-04-30T06:42:57.811ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "14650683-00cd-43c5-b0e9-fef6dc925522", "content-length": "169", "date": "Wed, 30 Apr 2025 06:42:57 GMT"} })
65962025-04-30T06:42:57.811ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/0/component/stage0/caboose?firmware_slot=0
65972025-04-30T06:42:57.815ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "0a43a8a4-e9e5-4417-9ca1-ad7aa16c21ea", "content-length": "178", "date": "Wed, 30 Apr 2025 06:42:57 GMT"} })
65982025-04-30T06:42:57.815ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/0/component/stage0/caboose?firmware_slot=1
65992025-04-30T06:42:57.819ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "87e7928e-c7c1-42c4-832a-ab196b5937ac", "content-length": "178", "date": "Wed, 30 Apr 2025 06:42:57 GMT"} })
66002025-04-30T06:42:57.819ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/0/component/rot/cmpa
66012025-04-30T06:42:57.820ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "78beefa8-7fa3-4ed0-94a0-81ead8ae80f8", "content-length": "702", "date": "Wed, 30 Apr 2025 06:42:57 GMT"} })
66022025-04-30T06:42:57.821ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/0/component/rot/cfpa
66032025-04-30T06:42:57.822ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "abc7bac7-cdc4-44d1-b2b9-717b4d68c88e", "content-length": "727", "date": "Wed, 30 Apr 2025 06:42:57 GMT"} })
66042025-04-30T06:42:57.823ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/0/component/rot/cfpa
66052025-04-30T06:42:57.824ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "218e3673-dfd9-42a1-84d8-30e13621974b", "content-length": "729", "date": "Wed, 30 Apr 2025 06:42:57 GMT"} })
66062025-04-30T06:42:57.824ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/0/component/rot/cfpa
66072025-04-30T06:42:57.826ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "2141b9f8-8150-4d94-a82d-56219c73dad6", "content-length": "728", "date": "Wed, 30 Apr 2025 06:42:57 GMT"} })
66082025-04-30T06:42:57.827ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/1
66092025-04-30T06:42:57.830ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "71a5c8b5-16f0-4c8d-86e4-418ec4ccdfa3", "content-length": "723", "date": "Wed, 30 Apr 2025 06:42:57 GMT"} })
66102025-04-30T06:42:57.830ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/1/component/sp/caboose?firmware_slot=0
66112025-04-30T06:42:57.832ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "cb505c8f-cd3e-4b4e-936e-57e0c145c4fb", "content-length": "109", "date": "Wed, 30 Apr 2025 06:42:57 GMT"} })
66122025-04-30T06:42:57.832ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/1/component/sp/caboose?firmware_slot=1
66132025-04-30T06:42:57.834ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "54ae7cde-b29d-4375-850e-8eb33e495641", "content-length": "109", "date": "Wed, 30 Apr 2025 06:42:57 GMT"} })
66142025-04-30T06:42:57.834ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/1/component/rot/caboose?firmware_slot=0
66152025-04-30T06:42:57.835ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
66162025-04-30T06:42:57.836ZTRACtest_snapshot_basic (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
66172025-04-30T06:42:57.836ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
66182025-04-30T06:42:57.836ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {} }
66192025-04-30T06:42:57.836ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
66202025-04-30T06:42:57.837ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "92bbc21f-0935-4c71-a606-226ea112f3cb", "content-length": "169", "date": "Wed, 30 Apr 2025 06:42:57 GMT"} })
66212025-04-30T06:42:57.837ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/1/component/rot/caboose?firmware_slot=1
66222025-04-30T06:42:57.840ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "a29d222f-c884-4101-baeb-46f2e5889983", "content-length": "169", "date": "Wed, 30 Apr 2025 06:42:57 GMT"} })
66232025-04-30T06:42:57.840ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/1/component/stage0/caboose?firmware_slot=0
66242025-04-30T06:42:57.842ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
66252025-04-30T06:42:57.850ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
66262025-04-30T06:42:57.851ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
66272025-04-30T06:42:57.851ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {} }
66282025-04-30T06:42:57.852ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
66292025-04-30T06:42:57.857ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
66302025-04-30T06:42:57.857ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {} }
66312025-04-30T06:42:57.857ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
66322025-04-30T06:42:57.857ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "1fe2f9db-91b0-43ee-bc9b-ff91339f53a0", "content-length": "178", "date": "Wed, 30 Apr 2025 06:42:57 GMT"} })
66332025-04-30T06:42:57.857ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/1/component/stage0/caboose?firmware_slot=1
66342025-04-30T06:42:57.861ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "dacd05ed-31eb-4743-a7fc-a53285ce7791", "content-length": "178", "date": "Wed, 30 Apr 2025 06:42:57 GMT"} })
66352025-04-30T06:42:57.861ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/1/component/rot/cmpa
66362025-04-30T06:42:57.862ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "3a713e77-c017-41c0-9af3-71eba6247e0c", "content-length": "702", "date": "Wed, 30 Apr 2025 06:42:57 GMT"} })
66372025-04-30T06:42:57.863ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/1/component/rot/cfpa
66382025-04-30T06:42:57.864ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "3bdd84bf-6f72-41a4-855f-80dd41bc5667", "content-length": "727", "date": "Wed, 30 Apr 2025 06:42:57 GMT"} })
66392025-04-30T06:42:57.865ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/1/component/rot/cfpa
66402025-04-30T06:42:57.867ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "ff202976-d37c-45a2-aed6-0f5cb0e1fab0", "content-length": "729", "date": "Wed, 30 Apr 2025 06:42:57 GMT"} })
66412025-04-30T06:42:57.867ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/1/component/rot/cfpa
66422025-04-30T06:42:57.870ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "59c8aa0e-6482-4f47-add0-af98fbbf98a5", "content-length": "728", "date": "Wed, 30 Apr 2025 06:42:57 GMT"} })
66432025-04-30T06:42:57.870ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
66442025-04-30T06:42:57.870ZTRACtest_snapshot_basic (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
66452025-04-30T06:42:57.870ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
66462025-04-30T06:42:57.870ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
66472025-04-30T06:42:57.871ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
66482025-04-30T06:42:57.885ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
66492025-04-30T06:42:57.889ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
66502025-04-30T06:42:57.889ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
66512025-04-30T06:42:57.889ZTRACtest_snapshot_basic (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
66522025-04-30T06:42:57.890ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
66532025-04-30T06:42:57.890ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
66542025-04-30T06:42:57.890ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
66552025-04-30T06:42:57.912ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
66562025-04-30T06:42:57.921ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
66572025-04-30T06:42:57.921ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
66582025-04-30T06:42:57.921ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
66592025-04-30T06:42:57.922ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
66602025-04-30T06:42:57.977ZDEBGtest_snapshot_basic (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:41923
66612025-04-30T06:42:57.977ZDEBGtest_snapshot_basic (BackgroundTasks): client request SledAgent = http://[::1]:41923 background_task = inventory_collection body = None method = GET uri = http://[::1]:41923/inventory
66622025-04-30T06:42:57.979ZINFOtest_snapshot_basic (dropshot): accepted connection local_addr = [::1]:41923 remote_addr = [::1]:62326 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
66632025-04-30T06:42:57.982ZDEBGtest_snapshot_basic: attempt to fetch latest inventory collection result = Ok("found")
66642025-04-30T06:42:57.982ZTRACtest_snapshot_basic (dropshot): incoming request local_addr = [::1]:41923 method = GET remote_addr = [::1]:62326 req_id = acfb2bfa-2f74-4a13-b923-bd36662c780d sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
66652025-04-30T06:42:57.983ZINFOtest_snapshot_basic (dropshot): request completed latency_us = 823 local_addr = [::1]:41923 method = GET remote_addr = [::1]:62326 req_id = acfb2bfa-2f74-4a13-b923-bd36662c780d response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
66662025-04-30T06:42:57.984ZDEBGtest_snapshot_basic (BackgroundTasks): client response SledAgent = http://[::1]:41923 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:41923/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "acfb2bfa-2f74-4a13-b923-bd36662c780d", "content-length": "3141", "date": "Wed, 30 Apr 2025 06:42:57 GMT"} })
66672025-04-30T06:42:57.986ZDEBGtest_snapshot_basic (BackgroundTasks): begin collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
66682025-04-30T06:42:57.986ZDEBGtest_snapshot_basic (BackgroundTasks): end collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
66692025-04-30T06:42:57.986ZDEBGtest_snapshot_basic (BackgroundTasks): finished collection background_task = inventory_collection
66702025-04-30T06:42:57.988ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
66712025-04-30T06:42:57.988ZTRACtest_snapshot_basic (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
66722025-04-30T06:42:57.988ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
66732025-04-30T06:42:57.988ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
66742025-04-30T06:42:57.989ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
66752025-04-30T06:42:57.993ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
66762025-04-30T06:42:57.997ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
66772025-04-30T06:42:57.998ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
66782025-04-30T06:42:57.998ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
66792025-04-30T06:42:57.998ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
66802025-04-30T06:42:58.033ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
66812025-04-30T06:42:58.033ZTRACtest_snapshot_basic (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
66822025-04-30T06:42:58.033ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
66832025-04-30T06:42:58.033ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {} }
66842025-04-30T06:42:58.034ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
66852025-04-30T06:42:58.035ZDEBGtest_snapshot_basic (results-sink): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
66862025-04-30T06:42:58.035ZDEBGtest_snapshot_basic (results-sink-cluster): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
66872025-04-30T06:42:58.036ZDEBGtest_snapshot_basic (clickhouse-client): unrolling 1 total samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 6ad06c6b-9a6e-4ed1-b3f4-5d2e9da3f8a7
66882025-04-30T06:42:58.037ZTRACtest_snapshot_basic (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 6ad06c6b-9a6e-4ed1-b3f4-5d2e9da3f8a7 n_columns = 4 n_rows = 2 sql = INSERT INTO oximeter.fields_string FORMAT Native
66892025-04-30T06:42:58.039ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
66902025-04-30T06:42:58.044ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
66912025-04-30T06:42:58.044ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
66922025-04-30T06:42:58.044ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {} }
66932025-04-30T06:42:58.045ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
66942025-04-30T06:42:58.047ZWARNtest_snapshot_basic (datastore): Non-retryable transaction failure err = DatabaseError(UniqueViolation, "duplicate key value violates unique constraint \\"inv_caboose_pkey\\"") transaction = inventory_insert_collection
66952025-04-30T06:42:58.047ZWARNtest_snapshot_basic (BackgroundTasks): inventory collection failed background_task = inventory_collection error = failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint "inv_caboose_pkey"
66962025-04-30T06:42:58.048ZDEBGtest_snapshot_basic (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 566.76918ms iteration = 19 status = {"error":"failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint \\"inv_caboose_pkey\\""}
66972025-04-30T06:42:58.048ZDEBGtest_snapshot_basic (BackgroundTasks): activating background_task = inventory_collection iteration = 20 reason = Signaled
66982025-04-30T06:42:58.048ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
66992025-04-30T06:42:58.048ZTRACtest_snapshot_basic (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
67002025-04-30T06:42:58.048ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
67012025-04-30T06:42:58.048ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
67022025-04-30T06:42:58.048ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
67032025-04-30T06:42:58.058ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
67042025-04-30T06:42:58.064ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
67052025-04-30T06:42:58.064ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
67062025-04-30T06:42:58.064ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
67072025-04-30T06:42:58.064ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
67082025-04-30T06:42:58.065ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
67092025-04-30T06:42:58.065ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {} }
67102025-04-30T06:42:58.066ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
67112025-04-30T06:42:58.067ZDEBGtest_snapshot_basic (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 6ad06c6b-9a6e-4ed1-b3f4-5d2e9da3f8a7 n_rows = 2 table_name = fields_string
67122025-04-30T06:42:58.067ZTRACtest_snapshot_basic (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 6ad06c6b-9a6e-4ed1-b3f4-5d2e9da3f8a7 n_columns = 4 n_rows = 1 sql = INSERT INTO oximeter.measurements_i64 FORMAT Native
67132025-04-30T06:42:58.069ZDEBGtest_snapshot_basic (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = [(d1ad44e5-18dd-4193-a70e-bf9042f0e0e1, 0)]
67142025-04-30T06:42:58.069ZTRACtest_snapshot_basic (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
67152025-04-30T06:42:58.070ZTRACtest_snapshot_basic (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: d3 28 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 .(..........._mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:54739 req_id = 7cb8a7dc-d489-49da-ad9a-1ffc33f79fe5
67162025-04-30T06:42:58.071ZDEBGtest_snapshot_basic (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 54056,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:54739 req_id = 7cb8a7dc-d489-49da-ad9a-1ffc33f79fe5
67172025-04-30T06:42:58.071ZDEBGtest_snapshot_basic (store): query key key = _mgs._tcp
67182025-04-30T06:42:58.071ZDEBGtest_snapshot_basic (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
67192025-04-30T06:42:58.071ZDEBGtest_snapshot_basic (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:54739 query = LowerQuery { name: LowerName(Name("_mgs._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 41502, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = 7cb8a7dc-d489-49da-ad9a-1ffc33f79fe5
67202025-04-30T06:42:58.072ZDEBGtest_snapshot_basic (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 41502, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 4242, tv_nsec: 871088884 } })
67212025-04-30T06:42:58.072ZTRACtest_snapshot_basic (DnsResolver): lookup_service_targets: looking up SRV target
67222025-04-30T06:42:58.074ZDEBGtest_snapshot_basic (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 6ad06c6b-9a6e-4ed1-b3f4-5d2e9da3f8a7 n_rows = 1 table_name = measurements_i64
67232025-04-30T06:42:58.074ZTRACtest_snapshot_basic (results-sink): successfully inserted samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
67242025-04-30T06:42:58.074ZTRACtest_snapshot_basic (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: ce d8 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e ............-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:62185 req_id = 32468179-1796-40f0-aa49-241713313f39
67252025-04-30T06:42:58.074ZDEBGtest_snapshot_basic (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 52952,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:62185 req_id = 32468179-1796-40f0-aa49-241713313f39
67262025-04-30T06:42:58.075ZDEBGtest_snapshot_basic (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
67272025-04-30T06:42:58.075ZDEBGtest_snapshot_basic (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:62185 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = 32468179-1796-40f0-aa49-241713313f39
67282025-04-30T06:42:58.122ZTRACtest_snapshot_basic (DnsResolver): lookup_all_socket_v6 srv dns_name = _clickhouse-admin-keeper._tcp.control-plane.oxide.internal
67292025-04-30T06:42:58.124ZTRACtest_snapshot_basic (dns): buffer buffer = Length: 87 (0x57) bytes\n0000: 4f 05 01 00 00 01 00 00 00 00 00 01 18 5f 63 6c O............_cl\n0010: 69 63 6b 68 6f 75 73 65 2d 61 64 6d 69 6e 2d 6b ickhouse-admin-k\n0020: 65 65 70 65 72 04 5f 74 63 70 0d 63 6f 6e 74 72 eeper._tcp.contr\n0030: 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 69 ol-plane.oxide.i\n0040: 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 04 nternal..!....).\n0050: d0 00 00 00 00 00 00 ....... kind = dns peer_addr = [::1]:52567 req_id = 9d97289d-36e2-49b0-a835-6c06ba14d5ea
67302025-04-30T06:42:58.128ZDEBGtest_snapshot_basic (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 20229,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 24,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 97,\n 100,\n 109,\n 105,\n 110,\n 45,\n 107,\n 101,\n 101,\n 112,\n 101,\n 114,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:52567 req_id = 9d97289d-36e2-49b0-a835-6c06ba14d5ea
67312025-04-30T06:42:58.129ZDEBGtest_snapshot_basic (store): query key key = _clickhouse-admin-keeper._tcp
67322025-04-30T06:42:58.129ZERROtest_snapshot_basic (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 20229, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } NXDOMAIN: no records found for name: "_clickhouse-admin-keeper._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:52567 req_id = 9d97289d-36e2-49b0-a835-6c06ba14d5ea
67332025-04-30T06:42:58.130ZDEBGtest_snapshot_basic (BackgroundTasks): begin collection background_task = inventory_collection
67342025-04-30T06:42:58.130ZDEBGtest_snapshot_basic (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:41502
67352025-04-30T06:42:58.130ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/ignition
67362025-04-30T06:42:58.132ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "af3422ed-5ce9-434a-a202-f6ff02d38258", "content-length": "835", "date": "Wed, 30 Apr 2025 06:42:57 GMT"} })
67372025-04-30T06:42:58.132ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/0
67382025-04-30T06:42:58.135ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "38870b5f-3438-4010-b7fd-1571d90241a1", "content-length": "734", "date": "Wed, 30 Apr 2025 06:42:57 GMT"} })
67392025-04-30T06:42:58.135ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/0/component/sp/caboose?firmware_slot=0
67402025-04-30T06:42:58.137ZWARNtest_snapshot_basic (results-sink-cluster): failed to insert some results into metric DB collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 error = Failed to check out connection to database: No backends found for this service file = oximeter/collector/src/results_sink.rs:92
67412025-04-30T06:42:58.138ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "1130db58-edd9-4e71-a7d3-bf74cc5a1096", "content-length": "111", "date": "Wed, 30 Apr 2025 06:42:57 GMT"} })
67422025-04-30T06:42:58.138ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/0/component/sp/caboose?firmware_slot=1
67432025-04-30T06:42:58.140ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "e0048a2f-e556-4568-b9fd-7e86f7fafe40", "content-length": "111", "date": "Wed, 30 Apr 2025 06:42:57 GMT"} })
67442025-04-30T06:42:58.140ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/0/component/rot/caboose?firmware_slot=0
67452025-04-30T06:42:58.142ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "db9ca0de-986b-479e-9df5-9e64075a4210", "content-length": "170", "date": "Wed, 30 Apr 2025 06:42:57 GMT"} })
67462025-04-30T06:42:58.142ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/0/component/rot/caboose?firmware_slot=1
67472025-04-30T06:42:58.144ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "89cde80a-38f5-4672-9558-583ff482b43b", "content-length": "170", "date": "Wed, 30 Apr 2025 06:42:57 GMT"} })
67482025-04-30T06:42:58.144ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/0/component/stage0/caboose?firmware_slot=0
67492025-04-30T06:42:58.148ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "c867c197-0207-4461-be04-e91dd98db8b1", "content-length": "179", "date": "Wed, 30 Apr 2025 06:42:57 GMT"} })
67502025-04-30T06:42:58.148ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/0/component/stage0/caboose?firmware_slot=1
67512025-04-30T06:42:58.148ZDEBGtest_snapshot_basic: attempt to fetch latest inventory collection result = Ok("found")
67522025-04-30T06:42:58.152ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "60b9a2a7-e651-458c-9162-00fe2a341327", "content-length": "179", "date": "Wed, 30 Apr 2025 06:42:57 GMT"} })
67532025-04-30T06:42:58.163ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/0/component/rot/cmpa
67542025-04-30T06:42:58.164ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "5088177b-0e7f-4021-8975-7732565d14b3", "content-length": "702", "date": "Wed, 30 Apr 2025 06:42:57 GMT"} })
67552025-04-30T06:42:58.164ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/0/component/rot/cfpa
67562025-04-30T06:42:58.166ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "80e1e94c-8844-4135-a771-79dd7b03f990", "content-length": "727", "date": "Wed, 30 Apr 2025 06:42:57 GMT"} })
67572025-04-30T06:42:58.166ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/0/component/rot/cfpa
67582025-04-30T06:42:58.167ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "fc344991-5b10-43c4-807c-d4627271d34a", "content-length": "729", "date": "Wed, 30 Apr 2025 06:42:57 GMT"} })
67592025-04-30T06:42:58.167ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/0/component/rot/cfpa
67602025-04-30T06:42:58.171ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "4f9c0421-c17a-4c02-8467-3d06ca930753", "content-length": "728", "date": "Wed, 30 Apr 2025 06:42:57 GMT"} })
67612025-04-30T06:42:58.171ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/1
67622025-04-30T06:42:58.180ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "efbed6b5-ef47-40c3-81db-d97cacd86fe9", "content-length": "734", "date": "Wed, 30 Apr 2025 06:42:57 GMT"} })
67632025-04-30T06:42:58.180ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/1/component/sp/caboose?firmware_slot=0
67642025-04-30T06:42:58.190ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "dab026f9-6e9f-4a7f-ad5e-6164bf42b5c9", "content-length": "111", "date": "Wed, 30 Apr 2025 06:42:57 GMT"} })
67652025-04-30T06:42:58.190ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/1/component/sp/caboose?firmware_slot=1
67662025-04-30T06:42:58.193ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "f4e22549-98d3-495d-8087-e2db0a8a8b2a", "content-length": "111", "date": "Wed, 30 Apr 2025 06:42:57 GMT"} })
67672025-04-30T06:42:58.194ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/1/component/rot/caboose?firmware_slot=0
67682025-04-30T06:42:58.197ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "99f27e0d-92d0-459b-9304-d038ab1ea65d", "content-length": "170", "date": "Wed, 30 Apr 2025 06:42:57 GMT"} })
67692025-04-30T06:42:58.197ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/1/component/rot/caboose?firmware_slot=1
67702025-04-30T06:42:58.198ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
67712025-04-30T06:42:58.198ZTRACtest_snapshot_basic (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
67722025-04-30T06:42:58.198ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
67732025-04-30T06:42:58.198ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {} }
67742025-04-30T06:42:58.200ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
67752025-04-30T06:42:58.202ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "27dee5b3-038f-4a65-997f-5aa2ae93cfd5", "content-length": "170", "date": "Wed, 30 Apr 2025 06:42:57 GMT"} })
67762025-04-30T06:42:58.203ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/1/component/stage0/caboose?firmware_slot=0
67772025-04-30T06:42:58.203ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
67782025-04-30T06:42:58.209ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
67792025-04-30T06:42:58.209ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
67802025-04-30T06:42:58.209ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {} }
67812025-04-30T06:42:58.210ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
67822025-04-30T06:42:58.215ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
67832025-04-30T06:42:58.215ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {} }
67842025-04-30T06:42:58.216ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
67852025-04-30T06:42:58.216ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "33393d60-28e7-46fa-9499-85fed51cf9b5", "content-length": "179", "date": "Wed, 30 Apr 2025 06:42:57 GMT"} })
67862025-04-30T06:42:58.216ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/1/component/stage0/caboose?firmware_slot=1
67872025-04-30T06:42:58.220ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "e81203bd-b6dc-47c2-a1bb-e5479bd14fed", "content-length": "179", "date": "Wed, 30 Apr 2025 06:42:57 GMT"} })
67882025-04-30T06:42:58.220ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/1/component/rot/cmpa
67892025-04-30T06:42:58.221ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "1eadae53-a6f2-4dba-99a8-4f2e31e93038", "content-length": "702", "date": "Wed, 30 Apr 2025 06:42:57 GMT"} })
67902025-04-30T06:42:58.221ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/1/component/rot/cfpa
67912025-04-30T06:42:58.223ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "609fc488-f677-418e-b037-b405b2dade96", "content-length": "727", "date": "Wed, 30 Apr 2025 06:42:57 GMT"} })
67922025-04-30T06:42:58.223ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/1/component/rot/cfpa
67932025-04-30T06:42:58.224ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "d38c2653-e367-41d6-8509-8ec0c7597daf", "content-length": "729", "date": "Wed, 30 Apr 2025 06:42:57 GMT"} })
67942025-04-30T06:42:58.224ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/1/component/rot/cfpa
67952025-04-30T06:42:58.225ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "303c705a-01db-4b24-8fe2-0d071a74e61b", "content-length": "728", "date": "Wed, 30 Apr 2025 06:42:57 GMT"} })
67962025-04-30T06:42:58.226ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/0
67972025-04-30T06:42:58.227ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "b9927c86-b10d-4ffc-8029-b2dbd2bcfc8a", "content-length": "723", "date": "Wed, 30 Apr 2025 06:42:57 GMT"} })
67982025-04-30T06:42:58.227ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/0/component/sp/caboose?firmware_slot=0
67992025-04-30T06:42:58.229ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "706af1ae-5a28-4226-b231-351f8112cbd6", "content-length": "109", "date": "Wed, 30 Apr 2025 06:42:57 GMT"} })
68002025-04-30T06:42:58.229ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/0/component/sp/caboose?firmware_slot=1
68012025-04-30T06:42:58.232ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "6330f90e-fc33-42a9-9dfd-cebe51063b76", "content-length": "109", "date": "Wed, 30 Apr 2025 06:42:57 GMT"} })
68022025-04-30T06:42:58.232ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/0/component/rot/caboose?firmware_slot=0
68032025-04-30T06:42:58.237ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "9d27ae47-dd78-471f-9138-737ec259347a", "content-length": "169", "date": "Wed, 30 Apr 2025 06:42:57 GMT"} })
68042025-04-30T06:42:58.238ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/0/component/rot/caboose?firmware_slot=1
68052025-04-30T06:42:58.240ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "ceb7f78f-6bdc-4ce7-afe3-e8e22ba2c01c", "content-length": "169", "date": "Wed, 30 Apr 2025 06:42:57 GMT"} })
68062025-04-30T06:42:58.240ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/0/component/stage0/caboose?firmware_slot=0
68072025-04-30T06:42:58.248ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "8db25187-ac92-4a36-b901-d9b9655662e8", "content-length": "178", "date": "Wed, 30 Apr 2025 06:42:57 GMT"} })
68082025-04-30T06:42:58.248ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/0/component/stage0/caboose?firmware_slot=1
68092025-04-30T06:42:58.254ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "9b2852d5-d648-49e6-af44-6e948d547c8c", "content-length": "178", "date": "Wed, 30 Apr 2025 06:42:57 GMT"} })
68102025-04-30T06:42:58.254ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/0/component/rot/cmpa
68112025-04-30T06:42:58.256ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "4bc52d20-1956-475e-a818-d1207357db53", "content-length": "702", "date": "Wed, 30 Apr 2025 06:42:57 GMT"} })
68122025-04-30T06:42:58.256ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/0/component/rot/cfpa
68132025-04-30T06:42:58.260ZDEBGtest_snapshot_basic: attempt to fetch latest inventory collection result = Ok("found")
68142025-04-30T06:42:58.260ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "500ca6be-8156-4a1f-8f3f-92cc01c84855", "content-length": "727", "date": "Wed, 30 Apr 2025 06:42:57 GMT"} })
68152025-04-30T06:42:58.261ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/0/component/rot/cfpa
68162025-04-30T06:42:58.262ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "687b5390-627a-40e1-a704-064751a3a9de", "content-length": "729", "date": "Wed, 30 Apr 2025 06:42:57 GMT"} })
68172025-04-30T06:42:58.263ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/0/component/rot/cfpa
68182025-04-30T06:42:58.263ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "d0fa832e-a07f-44a5-99a1-221c220ee1dd", "content-length": "728", "date": "Wed, 30 Apr 2025 06:42:57 GMT"} })
68192025-04-30T06:42:58.264ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/1
68202025-04-30T06:42:58.265ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "f89fafba-ea87-43c0-9dd3-031f8d617943", "content-length": "723", "date": "Wed, 30 Apr 2025 06:42:57 GMT"} })
68212025-04-30T06:42:58.265ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/1/component/sp/caboose?firmware_slot=0
68222025-04-30T06:42:58.267ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "9815e074-968e-4a43-98ba-f18984ea2f5e", "content-length": "109", "date": "Wed, 30 Apr 2025 06:42:57 GMT"} })
68232025-04-30T06:42:58.267ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/1/component/sp/caboose?firmware_slot=1
68242025-04-30T06:42:58.269ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "748eb557-6e9f-4672-958c-03d15129faa5", "content-length": "109", "date": "Wed, 30 Apr 2025 06:42:57 GMT"} })
68252025-04-30T06:42:58.269ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/1/component/rot/caboose?firmware_slot=0
68262025-04-30T06:42:58.271ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "8128c02f-16e4-4365-bc39-2b117b2fa2b8", "content-length": "169", "date": "Wed, 30 Apr 2025 06:42:57 GMT"} })
68272025-04-30T06:42:58.271ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/1/component/rot/caboose?firmware_slot=1
68282025-04-30T06:42:58.273ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "14ac6225-8827-4686-b92f-7efae2ed9dd9", "content-length": "169", "date": "Wed, 30 Apr 2025 06:42:57 GMT"} })
68292025-04-30T06:42:58.273ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/1/component/stage0/caboose?firmware_slot=0
68302025-04-30T06:42:58.276ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "c0d7bc1d-1aef-42b6-af9b-2bb9efef8187", "content-length": "178", "date": "Wed, 30 Apr 2025 06:42:57 GMT"} })
68312025-04-30T06:42:58.276ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/1/component/stage0/caboose?firmware_slot=1
68322025-04-30T06:42:58.278ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "210c48ec-2361-4b72-8c8a-13af976e2432", "content-length": "178", "date": "Wed, 30 Apr 2025 06:42:57 GMT"} })
68332025-04-30T06:42:58.278ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/1/component/rot/cmpa
68342025-04-30T06:42:58.279ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "a84cd1bd-63d2-4542-82d9-ad247472e087", "content-length": "702", "date": "Wed, 30 Apr 2025 06:42:57 GMT"} })
68352025-04-30T06:42:58.279ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/1/component/rot/cfpa
68362025-04-30T06:42:58.280ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "90860b31-adea-49be-b5d5-675e72bfe384", "content-length": "727", "date": "Wed, 30 Apr 2025 06:42:57 GMT"} })
68372025-04-30T06:42:58.280ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/1/component/rot/cfpa
68382025-04-30T06:42:58.281ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "1a73383b-6619-4e70-b0be-5e9a9389710d", "content-length": "729", "date": "Wed, 30 Apr 2025 06:42:57 GMT"} })
68392025-04-30T06:42:58.281ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/1/component/rot/cfpa
68402025-04-30T06:42:58.282ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "f10bfc50-662f-4286-83fc-439d01c731ce", "content-length": "728", "date": "Wed, 30 Apr 2025 06:42:57 GMT"} })
68412025-04-30T06:42:58.283ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
68422025-04-30T06:42:58.283ZTRACtest_snapshot_basic (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
68432025-04-30T06:42:58.283ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
68442025-04-30T06:42:58.283ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
68452025-04-30T06:42:58.284ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
68462025-04-30T06:42:58.286ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
68472025-04-30T06:42:58.293ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
68482025-04-30T06:42:58.293ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
68492025-04-30T06:42:58.293ZTRACtest_snapshot_basic (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
68502025-04-30T06:42:58.293ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
68512025-04-30T06:42:58.293ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
68522025-04-30T06:42:58.294ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
68532025-04-30T06:42:58.297ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
68542025-04-30T06:42:58.303ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
68552025-04-30T06:42:58.303ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
68562025-04-30T06:42:58.303ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
68572025-04-30T06:42:58.305ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
68582025-04-30T06:42:58.360ZDEBGtest_snapshot_basic (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:41923
68592025-04-30T06:42:58.361ZDEBGtest_snapshot_basic (BackgroundTasks): client request SledAgent = http://[::1]:41923 background_task = inventory_collection body = None method = GET uri = http://[::1]:41923/inventory
68602025-04-30T06:42:58.362ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
68612025-04-30T06:42:58.362ZTRACtest_snapshot_basic (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
68622025-04-30T06:42:58.362ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
68632025-04-30T06:42:58.362ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {} }
68642025-04-30T06:42:58.363ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
68652025-04-30T06:42:58.363ZINFOtest_snapshot_basic (dropshot): accepted connection local_addr = [::1]:41923 remote_addr = [::1]:35218 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
68662025-04-30T06:42:58.364ZTRACtest_snapshot_basic (dropshot): incoming request local_addr = [::1]:41923 method = GET remote_addr = [::1]:35218 req_id = 31d14732-ebce-4e20-9cf0-5e971229fe07 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
68672025-04-30T06:42:58.365ZINFOtest_snapshot_basic (dropshot): request completed latency_us = 588 local_addr = [::1]:41923 method = GET remote_addr = [::1]:35218 req_id = 31d14732-ebce-4e20-9cf0-5e971229fe07 response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
68682025-04-30T06:42:58.365ZDEBGtest_snapshot_basic (BackgroundTasks): client response SledAgent = http://[::1]:41923 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:41923/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "31d14732-ebce-4e20-9cf0-5e971229fe07", "content-length": "3141", "date": "Wed, 30 Apr 2025 06:42:57 GMT"} })
68692025-04-30T06:42:58.367ZDEBGtest_snapshot_basic (BackgroundTasks): begin collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
68702025-04-30T06:42:58.368ZDEBGtest_snapshot_basic (BackgroundTasks): end collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
68712025-04-30T06:42:58.368ZDEBGtest_snapshot_basic (BackgroundTasks): finished collection background_task = inventory_collection
68722025-04-30T06:42:58.369ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
68732025-04-30T06:42:58.369ZTRACtest_snapshot_basic (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
68742025-04-30T06:42:58.369ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
68752025-04-30T06:42:58.370ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
68762025-04-30T06:42:58.370ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
68772025-04-30T06:42:58.371ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
68782025-04-30T06:42:58.376ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
68792025-04-30T06:42:58.376ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
68802025-04-30T06:42:58.376ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {} }
68812025-04-30T06:42:58.376ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
68822025-04-30T06:42:58.379ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
68832025-04-30T06:42:58.384ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
68842025-04-30T06:42:58.385ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
68852025-04-30T06:42:58.385ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
68862025-04-30T06:42:58.394ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
68872025-04-30T06:42:58.396ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
68882025-04-30T06:42:58.396ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {} }
68892025-04-30T06:42:58.396ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
68902025-04-30T06:42:58.444ZDEBGtest_snapshot_basic: attempt to fetch latest inventory collection result = Ok("found")
68912025-04-30T06:42:58.444ZWARNtest_snapshot_basic (datastore): Non-retryable transaction failure err = DatabaseError(UniqueViolation, "duplicate key value violates unique constraint \\"inv_caboose_pkey\\"") transaction = inventory_insert_collection
68922025-04-30T06:42:58.445ZWARNtest_snapshot_basic (BackgroundTasks): inventory collection failed background_task = inventory_collection error = failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint "inv_caboose_pkey"
68932025-04-30T06:42:58.445ZDEBGtest_snapshot_basic (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 397.091389ms iteration = 20 status = {"error":"failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint \\"inv_caboose_pkey\\""}
68942025-04-30T06:42:58.445ZDEBGtest_snapshot_basic (BackgroundTasks): activating background_task = inventory_collection iteration = 21 reason = Signaled
68952025-04-30T06:42:58.445ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
68962025-04-30T06:42:58.445ZTRACtest_snapshot_basic (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
68972025-04-30T06:42:58.445ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
68982025-04-30T06:42:58.445ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
68992025-04-30T06:42:58.446ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
69002025-04-30T06:42:58.449ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
69012025-04-30T06:42:58.453ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
69022025-04-30T06:42:58.453ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
69032025-04-30T06:42:58.453ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
69042025-04-30T06:42:58.453ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
69052025-04-30T06:42:58.456ZDEBGtest_snapshot_basic (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = [(d1ad44e5-18dd-4193-a70e-bf9042f0e0e1, 0)]
69062025-04-30T06:42:58.456ZTRACtest_snapshot_basic (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
69072025-04-30T06:42:58.459ZTRACtest_snapshot_basic (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: 34 8f 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 4............_mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:63561 req_id = 4c888a2d-db3a-449a-b4e2-db18f6840ce3
69082025-04-30T06:42:58.459ZDEBGtest_snapshot_basic (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 13455,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:63561 req_id = 4c888a2d-db3a-449a-b4e2-db18f6840ce3
69092025-04-30T06:42:58.459ZDEBGtest_snapshot_basic (store): query key key = _mgs._tcp
69102025-04-30T06:42:58.459ZDEBGtest_snapshot_basic (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
69112025-04-30T06:42:58.459ZDEBGtest_snapshot_basic (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:63561 query = LowerQuery { name: LowerName(Name("_mgs._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 41502, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = 4c888a2d-db3a-449a-b4e2-db18f6840ce3
69122025-04-30T06:42:58.463ZDEBGtest_snapshot_basic (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 41502, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 4243, tv_nsec: 261273041 } })
69132025-04-30T06:42:58.463ZTRACtest_snapshot_basic (DnsResolver): lookup_service_targets: looking up SRV target
69142025-04-30T06:42:58.463ZTRACtest_snapshot_basic (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: 19 22 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e ."..........-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:50242 req_id = b35efe24-70bb-4de8-8448-4f9efa576a51
69152025-04-30T06:42:58.463ZDEBGtest_snapshot_basic (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 6434,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:50242 req_id = b35efe24-70bb-4de8-8448-4f9efa576a51
69162025-04-30T06:42:58.464ZDEBGtest_snapshot_basic (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
69172025-04-30T06:42:58.464ZDEBGtest_snapshot_basic (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:50242 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = b35efe24-70bb-4de8-8448-4f9efa576a51
69182025-04-30T06:42:58.535ZTRACtest_snapshot_basic (DnsResolver): lookup_all_socket_v6 srv dns_name = _clickhouse-admin-keeper._tcp.control-plane.oxide.internal
69192025-04-30T06:42:58.536ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
69202025-04-30T06:42:58.536ZTRACtest_snapshot_basic (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
69212025-04-30T06:42:58.536ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
69222025-04-30T06:42:58.536ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {} }
69232025-04-30T06:42:58.537ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
69242025-04-30T06:42:58.538ZTRACtest_snapshot_basic (dns): buffer buffer = Length: 87 (0x57) bytes\n0000: 4d 5f 01 00 00 01 00 00 00 00 00 01 18 5f 63 6c M_..........._cl\n0010: 69 63 6b 68 6f 75 73 65 2d 61 64 6d 69 6e 2d 6b ickhouse-admin-k\n0020: 65 65 70 65 72 04 5f 74 63 70 0d 63 6f 6e 74 72 eeper._tcp.contr\n0030: 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 69 ol-plane.oxide.i\n0040: 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 04 nternal..!....).\n0050: d0 00 00 00 00 00 00 ....... kind = dns peer_addr = [::1]:55525 req_id = 028c7278-e259-482c-81f1-5822a1a37f25
69252025-04-30T06:42:58.538ZDEBGtest_snapshot_basic (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 19807,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 24,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 97,\n 100,\n 109,\n 105,\n 110,\n 45,\n 107,\n 101,\n 101,\n 112,\n 101,\n 114,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:55525 req_id = 028c7278-e259-482c-81f1-5822a1a37f25
69262025-04-30T06:42:58.538ZDEBGtest_snapshot_basic (store): query key key = _clickhouse-admin-keeper._tcp
69272025-04-30T06:42:58.539ZERROtest_snapshot_basic (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 19807, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } NXDOMAIN: no records found for name: "_clickhouse-admin-keeper._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:55525 req_id = 028c7278-e259-482c-81f1-5822a1a37f25
69282025-04-30T06:42:58.544ZDEBGtest_snapshot_basic (BackgroundTasks): begin collection background_task = inventory_collection
69292025-04-30T06:42:58.544ZDEBGtest_snapshot_basic (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:41502
69302025-04-30T06:42:58.544ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/ignition
69312025-04-30T06:42:58.545ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
69322025-04-30T06:42:58.551ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
69332025-04-30T06:42:58.551ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
69342025-04-30T06:42:58.551ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {} }
69352025-04-30T06:42:58.552ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
69362025-04-30T06:42:58.567ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
69372025-04-30T06:42:58.567ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {} }
69382025-04-30T06:42:58.568ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
69392025-04-30T06:42:58.571ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "c58f2228-a86a-49cf-86fd-e82b6431b709", "content-length": "835", "date": "Wed, 30 Apr 2025 06:42:58 GMT"} })
69402025-04-30T06:42:58.572ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/0
69412025-04-30T06:42:58.581ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "064d3647-628b-4aa8-a1fd-d09ef71cbd29", "content-length": "734", "date": "Wed, 30 Apr 2025 06:42:58 GMT"} })
69422025-04-30T06:42:58.582ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/0/component/sp/caboose?firmware_slot=0
69432025-04-30T06:42:58.595ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "5136f600-c6c0-4457-adda-61e33b710125", "content-length": "111", "date": "Wed, 30 Apr 2025 06:42:58 GMT"} })
69442025-04-30T06:42:58.595ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/0/component/sp/caboose?firmware_slot=1
69452025-04-30T06:42:58.597ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "2fd1ba90-4f19-4089-ac6e-0aee286d1287", "content-length": "111", "date": "Wed, 30 Apr 2025 06:42:58 GMT"} })
69462025-04-30T06:42:58.597ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/0/component/rot/caboose?firmware_slot=0
69472025-04-30T06:42:58.600ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "d18a883e-4b60-4583-a7e9-e042d12cea4f", "content-length": "170", "date": "Wed, 30 Apr 2025 06:42:58 GMT"} })
69482025-04-30T06:42:58.600ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/0/component/rot/caboose?firmware_slot=1
69492025-04-30T06:42:58.604ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "50274e94-602e-4151-bcf1-645b692f0ad6", "content-length": "170", "date": "Wed, 30 Apr 2025 06:42:58 GMT"} })
69502025-04-30T06:42:58.604ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/0/component/stage0/caboose?firmware_slot=0
69512025-04-30T06:42:58.609ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "142ba104-b6ea-4df6-a34c-32c978c0c7f5", "content-length": "179", "date": "Wed, 30 Apr 2025 06:42:58 GMT"} })
69522025-04-30T06:42:58.610ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/0/component/stage0/caboose?firmware_slot=1
69532025-04-30T06:42:58.613ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "edf565df-9ff1-4a7b-9170-422078813324", "content-length": "179", "date": "Wed, 30 Apr 2025 06:42:58 GMT"} })
69542025-04-30T06:42:58.613ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/0/component/rot/cmpa
69552025-04-30T06:42:58.615ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "e5a3a72c-4c7a-4abe-b084-b7e076935836", "content-length": "702", "date": "Wed, 30 Apr 2025 06:42:58 GMT"} })
69562025-04-30T06:42:58.616ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/0/component/rot/cfpa
69572025-04-30T06:42:58.616ZDEBGtest_snapshot_basic: attempt to fetch latest inventory collection result = Ok("found")
69582025-04-30T06:42:58.618ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "be762686-c13d-46ba-a78f-0ea3531fae55", "content-length": "727", "date": "Wed, 30 Apr 2025 06:42:58 GMT"} })
69592025-04-30T06:42:58.618ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/0/component/rot/cfpa
69602025-04-30T06:42:58.619ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "b6794c0c-7521-47b2-9b34-61b6b6cfff0d", "content-length": "729", "date": "Wed, 30 Apr 2025 06:42:58 GMT"} })
69612025-04-30T06:42:58.619ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/0/component/rot/cfpa
69622025-04-30T06:42:58.620ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "d360b1a4-62a7-4954-b975-9f8c908506d8", "content-length": "728", "date": "Wed, 30 Apr 2025 06:42:58 GMT"} })
69632025-04-30T06:42:58.620ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/1
69642025-04-30T06:42:58.621ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "b651e850-1fa5-4f96-ae26-2116a3c132c6", "content-length": "734", "date": "Wed, 30 Apr 2025 06:42:58 GMT"} })
69652025-04-30T06:42:58.622ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/1/component/sp/caboose?firmware_slot=0
69662025-04-30T06:42:58.624ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "43439ae9-c21c-4300-a5fd-38e854aaeadd", "content-length": "111", "date": "Wed, 30 Apr 2025 06:42:58 GMT"} })
69672025-04-30T06:42:58.624ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/1/component/sp/caboose?firmware_slot=1
69682025-04-30T06:42:58.626ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "0d83d0f9-8cea-4b75-ac9d-6cf63b297362", "content-length": "111", "date": "Wed, 30 Apr 2025 06:42:58 GMT"} })
69692025-04-30T06:42:58.626ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/1/component/rot/caboose?firmware_slot=0
69702025-04-30T06:42:58.628ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "0c09446c-e530-4f7e-b23f-106e5127ba94", "content-length": "170", "date": "Wed, 30 Apr 2025 06:42:58 GMT"} })
69712025-04-30T06:42:58.628ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/1/component/rot/caboose?firmware_slot=1
69722025-04-30T06:42:58.630ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "35c8d0ea-5cbf-44d8-80b2-7d1ac2a8361f", "content-length": "170", "date": "Wed, 30 Apr 2025 06:42:58 GMT"} })
69732025-04-30T06:42:58.630ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/1/component/stage0/caboose?firmware_slot=0
69742025-04-30T06:42:58.632ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "0aff6b55-c7a5-4d5d-b146-25f89a66da16", "content-length": "179", "date": "Wed, 30 Apr 2025 06:42:58 GMT"} })
69752025-04-30T06:42:58.632ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/1/component/stage0/caboose?firmware_slot=1
69762025-04-30T06:42:58.634ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "06b6dc86-068d-46bb-bece-892148a7e5ea", "content-length": "179", "date": "Wed, 30 Apr 2025 06:42:58 GMT"} })
69772025-04-30T06:42:58.634ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/1/component/rot/cmpa
69782025-04-30T06:42:58.635ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "b2ecdac1-c5a0-440f-b89b-9cd2a7207526", "content-length": "702", "date": "Wed, 30 Apr 2025 06:42:58 GMT"} })
69792025-04-30T06:42:58.635ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/1/component/rot/cfpa
69802025-04-30T06:42:58.636ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "6b16221f-ed2b-43e7-90e4-192b0a546d1f", "content-length": "727", "date": "Wed, 30 Apr 2025 06:42:58 GMT"} })
69812025-04-30T06:42:58.636ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/1/component/rot/cfpa
69822025-04-30T06:42:58.637ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "a8f6ee82-5b95-425a-a5ad-2fcfda0c9836", "content-length": "729", "date": "Wed, 30 Apr 2025 06:42:58 GMT"} })
69832025-04-30T06:42:58.637ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/1/component/rot/cfpa
69842025-04-30T06:42:58.638ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "e9627a82-39cd-4c01-9917-020ef5f62b4f", "content-length": "728", "date": "Wed, 30 Apr 2025 06:42:58 GMT"} })
69852025-04-30T06:42:58.638ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/0
69862025-04-30T06:42:58.640ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "3f745151-b3b8-4821-9970-a1aa3d673918", "content-length": "723", "date": "Wed, 30 Apr 2025 06:42:58 GMT"} })
69872025-04-30T06:42:58.640ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/0/component/sp/caboose?firmware_slot=0
69882025-04-30T06:42:58.642ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "2fd1de3c-0ef4-4bb8-b76e-abb4855a5de7", "content-length": "109", "date": "Wed, 30 Apr 2025 06:42:58 GMT"} })
69892025-04-30T06:42:58.642ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/0/component/sp/caboose?firmware_slot=1
69902025-04-30T06:42:58.644ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "fc86d66b-3318-4ebe-b8f0-a8c066b0c5aa", "content-length": "109", "date": "Wed, 30 Apr 2025 06:42:58 GMT"} })
69912025-04-30T06:42:58.644ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/0/component/rot/caboose?firmware_slot=0
69922025-04-30T06:42:58.646ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "1b405dd6-8bed-40b8-8024-9f8421809d33", "content-length": "169", "date": "Wed, 30 Apr 2025 06:42:58 GMT"} })
69932025-04-30T06:42:58.646ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/0/component/rot/caboose?firmware_slot=1
69942025-04-30T06:42:58.648ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "eba2f17d-6065-4e80-8e9f-4dfecc08b6e9", "content-length": "169", "date": "Wed, 30 Apr 2025 06:42:58 GMT"} })
69952025-04-30T06:42:58.648ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/0/component/stage0/caboose?firmware_slot=0
69962025-04-30T06:42:58.650ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "eb7311d5-bf70-47c3-aec9-f8624f1bb01d", "content-length": "178", "date": "Wed, 30 Apr 2025 06:42:58 GMT"} })
69972025-04-30T06:42:58.650ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/0/component/stage0/caboose?firmware_slot=1
69982025-04-30T06:42:58.652ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "466c074b-5c09-45bd-8035-19ad5084a6ef", "content-length": "178", "date": "Wed, 30 Apr 2025 06:42:58 GMT"} })
69992025-04-30T06:42:58.652ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/0/component/rot/cmpa
70002025-04-30T06:42:58.653ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "67e8486e-27d8-4a5b-8ae4-aee2a383ddc7", "content-length": "702", "date": "Wed, 30 Apr 2025 06:42:58 GMT"} })
70012025-04-30T06:42:58.653ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/0/component/rot/cfpa
70022025-04-30T06:42:58.654ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "5f382b38-b70f-4fa6-9a12-6cb68368bf7a", "content-length": "727", "date": "Wed, 30 Apr 2025 06:42:58 GMT"} })
70032025-04-30T06:42:58.654ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/0/component/rot/cfpa
70042025-04-30T06:42:58.655ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "50e456e2-0ec3-41f7-bce8-aa1b4c568b3a", "content-length": "729", "date": "Wed, 30 Apr 2025 06:42:58 GMT"} })
70052025-04-30T06:42:58.655ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/0/component/rot/cfpa
70062025-04-30T06:42:58.656ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "82b5639d-4821-4f51-908c-eeb74e41f1ec", "content-length": "728", "date": "Wed, 30 Apr 2025 06:42:58 GMT"} })
70072025-04-30T06:42:58.656ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/1
70082025-04-30T06:42:58.658ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "5d59de6d-e3e1-4881-a26e-7f0fe64c1fb6", "content-length": "723", "date": "Wed, 30 Apr 2025 06:42:58 GMT"} })
70092025-04-30T06:42:58.658ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/1/component/sp/caboose?firmware_slot=0
70102025-04-30T06:42:58.660ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "875a2a8a-f797-4664-b6cd-3ceb8e74d189", "content-length": "109", "date": "Wed, 30 Apr 2025 06:42:58 GMT"} })
70112025-04-30T06:42:58.660ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/1/component/sp/caboose?firmware_slot=1
70122025-04-30T06:42:58.663ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "7ac1f12a-c166-4bc8-9616-348cf5fa6b82", "content-length": "109", "date": "Wed, 30 Apr 2025 06:42:58 GMT"} })
70132025-04-30T06:42:58.663ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/1/component/rot/caboose?firmware_slot=0
70142025-04-30T06:42:58.665ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "09146fb8-30a3-42f5-9239-ca918da61b55", "content-length": "169", "date": "Wed, 30 Apr 2025 06:42:58 GMT"} })
70152025-04-30T06:42:58.665ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/1/component/rot/caboose?firmware_slot=1
70162025-04-30T06:42:58.666ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
70172025-04-30T06:42:58.666ZTRACtest_snapshot_basic (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
70182025-04-30T06:42:58.667ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
70192025-04-30T06:42:58.667ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {} }
70202025-04-30T06:42:58.668ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
70212025-04-30T06:42:58.669ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "2ac5de8e-dfab-4684-b7a2-a29edfc2821c", "content-length": "169", "date": "Wed, 30 Apr 2025 06:42:58 GMT"} })
70222025-04-30T06:42:58.669ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/1/component/stage0/caboose?firmware_slot=0
70232025-04-30T06:42:58.671ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
70242025-04-30T06:42:58.688ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
70252025-04-30T06:42:58.689ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
70262025-04-30T06:42:58.689ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {} }
70272025-04-30T06:42:58.689ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
70282025-04-30T06:42:58.693ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
70292025-04-30T06:42:58.693ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {} }
70302025-04-30T06:42:58.693ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
70312025-04-30T06:42:58.698ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "b8f76180-bc68-4ccf-9a82-20c61dad9fb9", "content-length": "178", "date": "Wed, 30 Apr 2025 06:42:58 GMT"} })
70322025-04-30T06:42:58.698ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/1/component/stage0/caboose?firmware_slot=1
70332025-04-30T06:42:58.703ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "6a5e6730-53a2-48d5-9a6f-c01cf52d07d9", "content-length": "178", "date": "Wed, 30 Apr 2025 06:42:58 GMT"} })
70342025-04-30T06:42:58.703ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/1/component/rot/cmpa
70352025-04-30T06:42:58.705ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "b055427d-f460-4dc8-a361-9a832415d636", "content-length": "702", "date": "Wed, 30 Apr 2025 06:42:58 GMT"} })
70362025-04-30T06:42:58.705ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/1/component/rot/cfpa
70372025-04-30T06:42:58.708ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "e6bd47b9-627e-43c5-8dbd-a0d7d342142f", "content-length": "727", "date": "Wed, 30 Apr 2025 06:42:58 GMT"} })
70382025-04-30T06:42:58.708ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/1/component/rot/cfpa
70392025-04-30T06:42:58.710ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "2b05bd50-012c-4ad0-b1b7-5704226898ce", "content-length": "729", "date": "Wed, 30 Apr 2025 06:42:58 GMT"} })
70402025-04-30T06:42:58.710ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/1/component/rot/cfpa
70412025-04-30T06:42:58.711ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "b14c27b4-c818-46f3-bf90-f3bcfa6c31a7", "content-length": "728", "date": "Wed, 30 Apr 2025 06:42:58 GMT"} })
70422025-04-30T06:42:58.711ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
70432025-04-30T06:42:58.711ZTRACtest_snapshot_basic (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
70442025-04-30T06:42:58.711ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
70452025-04-30T06:42:58.711ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
70462025-04-30T06:42:58.713ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
70472025-04-30T06:42:58.719ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
70482025-04-30T06:42:58.724ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
70492025-04-30T06:42:58.724ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
70502025-04-30T06:42:58.724ZTRACtest_snapshot_basic (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
70512025-04-30T06:42:58.724ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
70522025-04-30T06:42:58.724ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
70532025-04-30T06:42:58.724ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
70542025-04-30T06:42:58.726ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
70552025-04-30T06:42:58.730ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
70562025-04-30T06:42:58.731ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
70572025-04-30T06:42:58.731ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
70582025-04-30T06:42:58.731ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
70592025-04-30T06:42:58.735ZDEBGtest_snapshot_basic: attempt to fetch latest inventory collection result = Ok("found")
70602025-04-30T06:42:58.813ZDEBGtest_snapshot_basic (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:41923
70612025-04-30T06:42:58.814ZDEBGtest_snapshot_basic (BackgroundTasks): client request SledAgent = http://[::1]:41923 background_task = inventory_collection body = None method = GET uri = http://[::1]:41923/inventory
70622025-04-30T06:42:58.815ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
70632025-04-30T06:42:58.815ZTRACtest_snapshot_basic (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
70642025-04-30T06:42:58.815ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
70652025-04-30T06:42:58.816ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {} }
70662025-04-30T06:42:58.816ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
70672025-04-30T06:42:58.817ZINFOtest_snapshot_basic (dropshot): accepted connection local_addr = [::1]:41923 remote_addr = [::1]:35449 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
70682025-04-30T06:42:58.817ZDEBGtest_snapshot_basic (collection-task): sent timer-based collection request to the collection task collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
70692025-04-30T06:42:58.817ZDEBGtest_snapshot_basic (collection-task): collecting from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
70702025-04-30T06:42:58.818ZTRACtest_snapshot_basic (dropshot): incoming request local_addr = [::1]:41923 method = GET remote_addr = [::1]:35449 req_id = 136eed29-68e1-4aff-b246-55699819dca8 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
70712025-04-30T06:42:58.820ZINFOtest_snapshot_basic (dropshot): request completed latency_us = 1806 local_addr = [::1]:41923 method = GET remote_addr = [::1]:35449 req_id = 136eed29-68e1-4aff-b246-55699819dca8 response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
70722025-04-30T06:42:58.821ZTRACtest_snapshot_basic (collection-task): sent collection request to producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
70732025-04-30T06:42:58.822ZDEBGtest_snapshot_basic (collection-task): collected results from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 n_results = 1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
70742025-04-30T06:42:58.822ZTRACtest_snapshot_basic (collection-task): forwarded results to main collection loop collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
70752025-04-30T06:42:58.822ZTRACtest_snapshot_basic (collection-task): top of inner collection loop, waiting for next request collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
70762025-04-30T06:42:58.822ZDEBGtest_snapshot_basic (BackgroundTasks): client response SledAgent = http://[::1]:41923 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:41923/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "136eed29-68e1-4aff-b246-55699819dca8", "content-length": "3141", "date": "Wed, 30 Apr 2025 06:42:58 GMT"} })
70772025-04-30T06:42:58.824ZDEBGtest_snapshot_basic (BackgroundTasks): begin collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
70782025-04-30T06:42:58.824ZDEBGtest_snapshot_basic (BackgroundTasks): end collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
70792025-04-30T06:42:58.824ZDEBGtest_snapshot_basic (BackgroundTasks): finished collection background_task = inventory_collection
70802025-04-30T06:42:58.827ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
70812025-04-30T06:42:58.827ZTRACtest_snapshot_basic (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
70822025-04-30T06:42:58.827ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
70832025-04-30T06:42:58.827ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
70842025-04-30T06:42:58.828ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
70852025-04-30T06:42:58.828ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
70862025-04-30T06:42:58.834ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
70872025-04-30T06:42:58.834ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
70882025-04-30T06:42:58.834ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {} }
70892025-04-30T06:42:58.835ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
70902025-04-30T06:42:58.844ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
70912025-04-30T06:42:58.844ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {} }
70922025-04-30T06:42:58.846ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
70932025-04-30T06:42:58.846ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
70942025-04-30T06:42:58.851ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
70952025-04-30T06:42:58.851ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
70962025-04-30T06:42:58.851ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
70972025-04-30T06:42:58.851ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
70982025-04-30T06:42:58.929ZDEBGtest_snapshot_basic: attempt to fetch latest inventory collection result = Ok("found")
70992025-04-30T06:42:58.929ZWARNtest_snapshot_basic (datastore): Non-retryable transaction failure err = DatabaseError(UniqueViolation, "duplicate key value violates unique constraint \\"inv_caboose_pkey\\"") transaction = inventory_insert_collection
71002025-04-30T06:42:58.930ZWARNtest_snapshot_basic (BackgroundTasks): inventory collection failed background_task = inventory_collection error = failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint "inv_caboose_pkey"
71012025-04-30T06:42:58.930ZDEBGtest_snapshot_basic (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 485.144699ms iteration = 21 status = {"error":"failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint \\"inv_caboose_pkey\\""}
71022025-04-30T06:42:58.930ZDEBGtest_snapshot_basic (BackgroundTasks): activating background_task = inventory_collection iteration = 22 reason = Signaled
71032025-04-30T06:42:58.930ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
71042025-04-30T06:42:58.930ZTRACtest_snapshot_basic (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
71052025-04-30T06:42:58.930ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
71062025-04-30T06:42:58.930ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
71072025-04-30T06:42:58.931ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
71082025-04-30T06:42:58.951ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
71092025-04-30T06:42:58.955ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
71102025-04-30T06:42:58.956ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
71112025-04-30T06:42:58.956ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
71122025-04-30T06:42:58.957ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
71132025-04-30T06:42:58.964ZDEBGtest_snapshot_basic (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = [(d1ad44e5-18dd-4193-a70e-bf9042f0e0e1, 0)]
71142025-04-30T06:42:58.964ZTRACtest_snapshot_basic (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
71152025-04-30T06:42:58.965ZTRACtest_snapshot_basic (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: 0f e0 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 ............._mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:60904 req_id = b2013b45-cf9a-4269-89f8-9dd224785240
71162025-04-30T06:42:58.965ZDEBGtest_snapshot_basic (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 4064,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:60904 req_id = b2013b45-cf9a-4269-89f8-9dd224785240
71172025-04-30T06:42:58.965ZDEBGtest_snapshot_basic (store): query key key = _mgs._tcp
71182025-04-30T06:42:58.965ZDEBGtest_snapshot_basic (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
71192025-04-30T06:42:58.965ZDEBGtest_snapshot_basic (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:60904 query = LowerQuery { name: LowerName(Name("_mgs._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 41502, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = b2013b45-cf9a-4269-89f8-9dd224785240
71202025-04-30T06:42:58.966ZDEBGtest_snapshot_basic (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 41502, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 4243, tv_nsec: 765060985 } })
71212025-04-30T06:42:58.966ZTRACtest_snapshot_basic (DnsResolver): lookup_service_targets: looking up SRV target
71222025-04-30T06:42:58.966ZTRACtest_snapshot_basic (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: ab 67 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e .g..........-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:65322 req_id = deae1b38-6ceb-483c-bc73-748a6a7cf45b
71232025-04-30T06:42:58.966ZDEBGtest_snapshot_basic (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 43879,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:65322 req_id = deae1b38-6ceb-483c-bc73-748a6a7cf45b
71242025-04-30T06:42:58.966ZDEBGtest_snapshot_basic (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
71252025-04-30T06:42:58.966ZDEBGtest_snapshot_basic (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:65322 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = deae1b38-6ceb-483c-bc73-748a6a7cf45b
71262025-04-30T06:42:59.017ZTRACtest_snapshot_basic (DnsResolver): lookup_all_socket_v6 srv dns_name = _clickhouse-admin-keeper._tcp.control-plane.oxide.internal
71272025-04-30T06:42:59.018ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
71282025-04-30T06:42:59.018ZTRACtest_snapshot_basic (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
71292025-04-30T06:42:59.018ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
71302025-04-30T06:42:59.018ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {} }
71312025-04-30T06:42:59.018ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
71322025-04-30T06:42:59.019ZTRACtest_snapshot_basic (dns): buffer buffer = Length: 87 (0x57) bytes\n0000: 24 9d 01 00 00 01 00 00 00 00 00 01 18 5f 63 6c $............_cl\n0010: 69 63 6b 68 6f 75 73 65 2d 61 64 6d 69 6e 2d 6b ickhouse-admin-k\n0020: 65 65 70 65 72 04 5f 74 63 70 0d 63 6f 6e 74 72 eeper._tcp.contr\n0030: 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 69 ol-plane.oxide.i\n0040: 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 04 nternal..!....).\n0050: d0 00 00 00 00 00 00 ....... kind = dns peer_addr = [::1]:61404 req_id = fd1ff4e6-a6eb-4d04-a499-a711f51929a8
71332025-04-30T06:42:59.019ZDEBGtest_snapshot_basic (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 9373,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 24,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 97,\n 100,\n 109,\n 105,\n 110,\n 45,\n 107,\n 101,\n 101,\n 112,\n 101,\n 114,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:61404 req_id = fd1ff4e6-a6eb-4d04-a499-a711f51929a8
71342025-04-30T06:42:59.020ZDEBGtest_snapshot_basic (store): query key key = _clickhouse-admin-keeper._tcp
71352025-04-30T06:42:59.020ZERROtest_snapshot_basic (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 9373, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } NXDOMAIN: no records found for name: "_clickhouse-admin-keeper._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:61404 req_id = fd1ff4e6-a6eb-4d04-a499-a711f51929a8
71362025-04-30T06:42:59.020ZDEBGtest_snapshot_basic (BackgroundTasks): begin collection background_task = inventory_collection
71372025-04-30T06:42:59.020ZDEBGtest_snapshot_basic (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:41502
71382025-04-30T06:42:59.020ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/ignition
71392025-04-30T06:42:59.022ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
71402025-04-30T06:42:59.028ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
71412025-04-30T06:42:59.028ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
71422025-04-30T06:42:59.028ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {} }
71432025-04-30T06:42:59.028ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
71442025-04-30T06:42:59.030ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "732e3858-33b5-4c3d-8e03-62888d9f0a41", "content-length": "835", "date": "Wed, 30 Apr 2025 06:42:58 GMT"} })
71452025-04-30T06:42:59.031ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/0
71462025-04-30T06:42:59.032ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
71472025-04-30T06:42:59.032ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {} }
71482025-04-30T06:42:59.033ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
71492025-04-30T06:42:59.035ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "2bcb0dc8-69db-4fb2-9b30-193c39667186", "content-length": "734", "date": "Wed, 30 Apr 2025 06:42:58 GMT"} })
71502025-04-30T06:42:59.035ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/0/component/sp/caboose?firmware_slot=0
71512025-04-30T06:42:59.036ZDEBGtest_snapshot_basic (results-sink): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
71522025-04-30T06:42:59.036ZDEBGtest_snapshot_basic (results-sink-cluster): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
71532025-04-30T06:42:59.036ZDEBGtest_snapshot_basic (clickhouse-client): unrolling 1 total samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 6ad06c6b-9a6e-4ed1-b3f4-5d2e9da3f8a7
71542025-04-30T06:42:59.037ZTRACtest_snapshot_basic (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 6ad06c6b-9a6e-4ed1-b3f4-5d2e9da3f8a7 n_columns = 4 n_rows = 2 sql = INSERT INTO oximeter.fields_string FORMAT Native
71552025-04-30T06:42:59.043ZDEBGtest_snapshot_basic (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 6ad06c6b-9a6e-4ed1-b3f4-5d2e9da3f8a7 n_rows = 2 table_name = fields_string
71562025-04-30T06:42:59.043ZTRACtest_snapshot_basic (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 6ad06c6b-9a6e-4ed1-b3f4-5d2e9da3f8a7 n_columns = 4 n_rows = 1 sql = INSERT INTO oximeter.measurements_i64 FORMAT Native
71572025-04-30T06:42:59.049ZDEBGtest_snapshot_basic (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 6ad06c6b-9a6e-4ed1-b3f4-5d2e9da3f8a7 n_rows = 1 table_name = measurements_i64
71582025-04-30T06:42:59.049ZTRACtest_snapshot_basic (results-sink): successfully inserted samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
71592025-04-30T06:42:59.054ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "14bc4b31-22c2-4edb-b622-222836e1a536", "content-length": "111", "date": "Wed, 30 Apr 2025 06:42:58 GMT"} })
71602025-04-30T06:42:59.054ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/0/component/sp/caboose?firmware_slot=1
71612025-04-30T06:42:59.063ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "49cc9ce7-3359-4c52-b367-8ac9eeb1c8b3", "content-length": "111", "date": "Wed, 30 Apr 2025 06:42:58 GMT"} })
71622025-04-30T06:42:59.063ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/0/component/rot/caboose?firmware_slot=0
71632025-04-30T06:42:59.065ZDEBGtest_snapshot_basic: attempt to fetch latest inventory collection result = Ok("found")
71642025-04-30T06:42:59.069ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "987bcdc5-01ab-456b-a313-2fb5cfedba70", "content-length": "170", "date": "Wed, 30 Apr 2025 06:42:58 GMT"} })
71652025-04-30T06:42:59.069ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/0/component/rot/caboose?firmware_slot=1
71662025-04-30T06:42:59.074ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "c626bdab-304b-4afd-b37d-8ca39a928c9a", "content-length": "170", "date": "Wed, 30 Apr 2025 06:42:58 GMT"} })
71672025-04-30T06:42:59.074ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/0/component/stage0/caboose?firmware_slot=0
71682025-04-30T06:42:59.078ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "38c792c8-e4b8-4994-8b73-cc2d56d46794", "content-length": "179", "date": "Wed, 30 Apr 2025 06:42:58 GMT"} })
71692025-04-30T06:42:59.080ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/0/component/stage0/caboose?firmware_slot=1
71702025-04-30T06:42:59.084ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "773e39f5-b89c-4b8c-a36b-82297caf07c2", "content-length": "179", "date": "Wed, 30 Apr 2025 06:42:58 GMT"} })
71712025-04-30T06:42:59.084ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/0/component/rot/cmpa
71722025-04-30T06:42:59.085ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "26b57536-41af-404f-9445-da4a6cfb8406", "content-length": "702", "date": "Wed, 30 Apr 2025 06:42:58 GMT"} })
71732025-04-30T06:42:59.085ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/0/component/rot/cfpa
71742025-04-30T06:42:59.087ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "d0680ff8-1432-4ede-8e3b-0c7a3d20b33c", "content-length": "727", "date": "Wed, 30 Apr 2025 06:42:58 GMT"} })
71752025-04-30T06:42:59.087ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/0/component/rot/cfpa
71762025-04-30T06:42:59.089ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "dd7356b3-0f58-4a55-8f1d-4ee3d3137d9c", "content-length": "729", "date": "Wed, 30 Apr 2025 06:42:58 GMT"} })
71772025-04-30T06:42:59.089ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/0/component/rot/cfpa
71782025-04-30T06:42:59.091ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "aec31f33-63d0-492a-b076-ed7803f88baa", "content-length": "728", "date": "Wed, 30 Apr 2025 06:42:58 GMT"} })
71792025-04-30T06:42:59.091ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/1
71802025-04-30T06:42:59.092ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "9351d055-309a-48e4-b956-ad82c1c67e2f", "content-length": "734", "date": "Wed, 30 Apr 2025 06:42:58 GMT"} })
71812025-04-30T06:42:59.093ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/1/component/sp/caboose?firmware_slot=0
71822025-04-30T06:42:59.098ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "de892320-d4c8-4696-822b-6220a753c9e0", "content-length": "111", "date": "Wed, 30 Apr 2025 06:42:58 GMT"} })
71832025-04-30T06:42:59.098ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/1/component/sp/caboose?firmware_slot=1
71842025-04-30T06:42:59.111ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "160b5f83-df82-411a-93f5-376e26140bd5", "content-length": "111", "date": "Wed, 30 Apr 2025 06:42:58 GMT"} })
71852025-04-30T06:42:59.111ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/1/component/rot/caboose?firmware_slot=0
71862025-04-30T06:42:59.111ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "ab54d7ca-8aac-4291-8e50-fda222e898fd", "content-length": "170", "date": "Wed, 30 Apr 2025 06:42:58 GMT"} })
71872025-04-30T06:42:59.111ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/1/component/rot/caboose?firmware_slot=1
71882025-04-30T06:42:59.111ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "28cb02c6-535e-46d3-8158-ade766a4686b", "content-length": "170", "date": "Wed, 30 Apr 2025 06:42:58 GMT"} })
71892025-04-30T06:42:59.111ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/1/component/stage0/caboose?firmware_slot=0
71902025-04-30T06:42:59.112ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "999b85e2-ffbb-4bee-a30a-694c87988c92", "content-length": "179", "date": "Wed, 30 Apr 2025 06:42:58 GMT"} })
71912025-04-30T06:42:59.112ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/1/component/stage0/caboose?firmware_slot=1
71922025-04-30T06:42:59.115ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "3ea2e011-63c4-4b65-87c4-73698e87373f", "content-length": "179", "date": "Wed, 30 Apr 2025 06:42:58 GMT"} })
71932025-04-30T06:42:59.115ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/1/component/rot/cmpa
71942025-04-30T06:42:59.117ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
71952025-04-30T06:42:59.117ZTRACtest_snapshot_basic (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
71962025-04-30T06:42:59.117ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
71972025-04-30T06:42:59.117ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {} }
71982025-04-30T06:42:59.118ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
71992025-04-30T06:42:59.119ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "9f1abbfa-e76d-46ae-b7d8-32a9ee1cc662", "content-length": "702", "date": "Wed, 30 Apr 2025 06:42:58 GMT"} })
72002025-04-30T06:42:59.119ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/1/component/rot/cfpa
72012025-04-30T06:42:59.120ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "26fb67b0-feb4-4489-bb6b-523fa802de79", "content-length": "727", "date": "Wed, 30 Apr 2025 06:42:58 GMT"} })
72022025-04-30T06:42:59.120ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/1/component/rot/cfpa
72032025-04-30T06:42:59.122ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "3ffa7d0d-5dea-4d2f-b7ed-a6fe0404f28a", "content-length": "729", "date": "Wed, 30 Apr 2025 06:42:58 GMT"} })
72042025-04-30T06:42:59.122ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/1/component/rot/cfpa
72052025-04-30T06:42:59.132ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
72062025-04-30T06:42:59.143ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
72072025-04-30T06:42:59.143ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
72082025-04-30T06:42:59.143ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {} }
72092025-04-30T06:42:59.144ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
72102025-04-30T06:42:59.145ZWARNtest_snapshot_basic (results-sink-cluster): failed to insert some results into metric DB collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 error = Failed to check out connection to database: No backends found for this service file = oximeter/collector/src/results_sink.rs:92
72112025-04-30T06:42:59.146ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "8b4f389d-f392-49d4-bde4-5110e5dd1fe2", "content-length": "728", "date": "Wed, 30 Apr 2025 06:42:58 GMT"} })
72122025-04-30T06:42:59.146ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/0
72132025-04-30T06:42:59.148ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
72142025-04-30T06:42:59.148ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {} }
72152025-04-30T06:42:59.148ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
72162025-04-30T06:42:59.150ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "e74dbd4f-8960-470f-ba07-3d04e57c430d", "content-length": "723", "date": "Wed, 30 Apr 2025 06:42:58 GMT"} })
72172025-04-30T06:42:59.150ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/0/component/sp/caboose?firmware_slot=0
72182025-04-30T06:42:59.155ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "247e4bf4-9f6d-43d2-b8e4-655313b8d2b9", "content-length": "109", "date": "Wed, 30 Apr 2025 06:42:58 GMT"} })
72192025-04-30T06:42:59.155ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/0/component/sp/caboose?firmware_slot=1
72202025-04-30T06:42:59.161ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "50c02f12-005c-4cd0-889f-f75e809a9b5a", "content-length": "109", "date": "Wed, 30 Apr 2025 06:42:58 GMT"} })
72212025-04-30T06:42:59.162ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/0/component/rot/caboose?firmware_slot=0
72222025-04-30T06:42:59.164ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "87fd314d-c3aa-424f-99c3-61b84e63f220", "content-length": "169", "date": "Wed, 30 Apr 2025 06:42:58 GMT"} })
72232025-04-30T06:42:59.165ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/0/component/rot/caboose?firmware_slot=1
72242025-04-30T06:42:59.172ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "6bf995cb-cf3d-47a8-b87b-4f66ce0c1bfc", "content-length": "169", "date": "Wed, 30 Apr 2025 06:42:58 GMT"} })
72252025-04-30T06:42:59.172ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/0/component/stage0/caboose?firmware_slot=0
72262025-04-30T06:42:59.177ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "59b16581-1f44-4742-aa63-0f843eb29e4b", "content-length": "178", "date": "Wed, 30 Apr 2025 06:42:58 GMT"} })
72272025-04-30T06:42:59.177ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/0/component/stage0/caboose?firmware_slot=1
72282025-04-30T06:42:59.182ZDEBGtest_snapshot_basic: attempt to fetch latest inventory collection result = Ok("found")
72292025-04-30T06:42:59.184ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "ad05a595-7f83-4aae-9742-d048ff91ea22", "content-length": "178", "date": "Wed, 30 Apr 2025 06:42:58 GMT"} })
72302025-04-30T06:42:59.184ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/0/component/rot/cmpa
72312025-04-30T06:42:59.185ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "c494eadb-1ad0-4b89-a466-0c2bf440b49f", "content-length": "702", "date": "Wed, 30 Apr 2025 06:42:58 GMT"} })
72322025-04-30T06:42:59.185ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/0/component/rot/cfpa
72332025-04-30T06:42:59.186ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "78390bcf-0ab5-4e02-817c-b579e3223954", "content-length": "727", "date": "Wed, 30 Apr 2025 06:42:58 GMT"} })
72342025-04-30T06:42:59.186ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/0/component/rot/cfpa
72352025-04-30T06:42:59.189ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "deff70e3-876c-49ad-bd24-170b5c1252d7", "content-length": "729", "date": "Wed, 30 Apr 2025 06:42:58 GMT"} })
72362025-04-30T06:42:59.189ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/0/component/rot/cfpa
72372025-04-30T06:42:59.189ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "f5488bd2-d15b-403d-9b72-a279cc5cc96c", "content-length": "728", "date": "Wed, 30 Apr 2025 06:42:58 GMT"} })
72382025-04-30T06:42:59.189ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/1
72392025-04-30T06:42:59.189ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "652c574e-e01c-4079-9875-cd9ff666ccd3", "content-length": "723", "date": "Wed, 30 Apr 2025 06:42:58 GMT"} })
72402025-04-30T06:42:59.190ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/1/component/sp/caboose?firmware_slot=0
72412025-04-30T06:42:59.193ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "3c75c425-0546-4fad-acf6-96f7c24d3319", "content-length": "109", "date": "Wed, 30 Apr 2025 06:42:58 GMT"} })
72422025-04-30T06:42:59.193ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/1/component/sp/caboose?firmware_slot=1
72432025-04-30T06:42:59.194ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "9adb11db-5f5b-4bff-9f4a-08048a9cd060", "content-length": "109", "date": "Wed, 30 Apr 2025 06:42:58 GMT"} })
72442025-04-30T06:42:59.194ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/1/component/rot/caboose?firmware_slot=0
72452025-04-30T06:42:59.197ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "d8ad42e9-f7fc-43fb-8dd7-3a9a37e74ec6", "content-length": "169", "date": "Wed, 30 Apr 2025 06:42:58 GMT"} })
72462025-04-30T06:42:59.197ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/1/component/rot/caboose?firmware_slot=1
72472025-04-30T06:42:59.200ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "8424da6f-ef48-448f-8119-e402fcd40a38", "content-length": "169", "date": "Wed, 30 Apr 2025 06:42:58 GMT"} })
72482025-04-30T06:42:59.200ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/1/component/stage0/caboose?firmware_slot=0
72492025-04-30T06:42:59.202ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "17291c63-6b71-4610-af80-cd61eef96724", "content-length": "178", "date": "Wed, 30 Apr 2025 06:42:58 GMT"} })
72502025-04-30T06:42:59.202ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/1/component/stage0/caboose?firmware_slot=1
72512025-04-30T06:42:59.204ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "c5a9fef6-58ea-495a-8cc2-2626d98f00f8", "content-length": "178", "date": "Wed, 30 Apr 2025 06:42:58 GMT"} })
72522025-04-30T06:42:59.204ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/1/component/rot/cmpa
72532025-04-30T06:42:59.206ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "6b4b1c72-e8f1-4743-a41b-465a7625a569", "content-length": "702", "date": "Wed, 30 Apr 2025 06:42:58 GMT"} })
72542025-04-30T06:42:59.206ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/1/component/rot/cfpa
72552025-04-30T06:42:59.208ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "3ed623ce-389b-4057-9e51-8872fbbbc197", "content-length": "727", "date": "Wed, 30 Apr 2025 06:42:58 GMT"} })
72562025-04-30T06:42:59.208ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/1/component/rot/cfpa
72572025-04-30T06:42:59.209ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "a3569297-480f-4dc4-9bf6-3da773bd3552", "content-length": "729", "date": "Wed, 30 Apr 2025 06:42:58 GMT"} })
72582025-04-30T06:42:59.209ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/1/component/rot/cfpa
72592025-04-30T06:42:59.211ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "935b6efd-9950-4114-9870-91fe840bc75f", "content-length": "728", "date": "Wed, 30 Apr 2025 06:42:58 GMT"} })
72602025-04-30T06:42:59.211ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
72612025-04-30T06:42:59.211ZTRACtest_snapshot_basic (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
72622025-04-30T06:42:59.211ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
72632025-04-30T06:42:59.211ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
72642025-04-30T06:42:59.213ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
72652025-04-30T06:42:59.216ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
72662025-04-30T06:42:59.221ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
72672025-04-30T06:42:59.221ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
72682025-04-30T06:42:59.222ZTRACtest_snapshot_basic (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
72692025-04-30T06:42:59.222ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
72702025-04-30T06:42:59.222ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
72712025-04-30T06:42:59.222ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
72722025-04-30T06:42:59.228ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
72732025-04-30T06:42:59.233ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
72742025-04-30T06:42:59.233ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
72752025-04-30T06:42:59.233ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
72762025-04-30T06:42:59.234ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
72772025-04-30T06:42:59.234ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
72782025-04-30T06:42:59.234ZTRACtest_snapshot_basic (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
72792025-04-30T06:42:59.234ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
72802025-04-30T06:42:59.234ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {} }
72812025-04-30T06:42:59.235ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
72822025-04-30T06:42:59.294ZDEBGtest_snapshot_basic (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:41923
72832025-04-30T06:42:59.294ZDEBGtest_snapshot_basic (BackgroundTasks): client request SledAgent = http://[::1]:41923 background_task = inventory_collection body = None method = GET uri = http://[::1]:41923/inventory
72842025-04-30T06:42:59.295ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
72852025-04-30T06:42:59.305ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
72862025-04-30T06:42:59.305ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
72872025-04-30T06:42:59.305ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {} }
72882025-04-30T06:42:59.306ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
72892025-04-30T06:42:59.306ZINFOtest_snapshot_basic (dropshot): accepted connection local_addr = [::1]:41923 remote_addr = [::1]:53653 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
72902025-04-30T06:42:59.313ZTRACtest_snapshot_basic (dropshot): incoming request local_addr = [::1]:41923 method = GET remote_addr = [::1]:53653 req_id = cb41672d-a047-49ef-bd69-61f7d35ca28c sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
72912025-04-30T06:42:59.313ZINFOtest_snapshot_basic (dropshot): request completed latency_us = 716 local_addr = [::1]:41923 method = GET remote_addr = [::1]:53653 req_id = cb41672d-a047-49ef-bd69-61f7d35ca28c response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
72922025-04-30T06:42:59.314ZDEBGtest_snapshot_basic (BackgroundTasks): client response SledAgent = http://[::1]:41923 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:41923/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "cb41672d-a047-49ef-bd69-61f7d35ca28c", "content-length": "3141", "date": "Wed, 30 Apr 2025 06:42:58 GMT"} })
72932025-04-30T06:42:59.317ZDEBGtest_snapshot_basic (BackgroundTasks): begin collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
72942025-04-30T06:42:59.317ZDEBGtest_snapshot_basic (BackgroundTasks): end collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
72952025-04-30T06:42:59.317ZDEBGtest_snapshot_basic (BackgroundTasks): finished collection background_task = inventory_collection
72962025-04-30T06:42:59.319ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
72972025-04-30T06:42:59.319ZTRACtest_snapshot_basic (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
72982025-04-30T06:42:59.319ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
72992025-04-30T06:42:59.319ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
73002025-04-30T06:42:59.320ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
73012025-04-30T06:42:59.320ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
73022025-04-30T06:42:59.320ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {} }
73032025-04-30T06:42:59.321ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
73042025-04-30T06:42:59.324ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
73052025-04-30T06:42:59.330ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
73062025-04-30T06:42:59.330ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
73072025-04-30T06:42:59.330ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
73082025-04-30T06:42:59.331ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
73092025-04-30T06:42:59.388ZWARNtest_snapshot_basic (datastore): Non-retryable transaction failure err = DatabaseError(UniqueViolation, "duplicate key value violates unique constraint \\"inv_caboose_pkey\\"") transaction = inventory_insert_collection
73102025-04-30T06:42:59.388ZWARNtest_snapshot_basic (BackgroundTasks): inventory collection failed background_task = inventory_collection error = failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint "inv_caboose_pkey"
73112025-04-30T06:42:59.388ZDEBGtest_snapshot_basic (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 458.593548ms iteration = 22 status = {"error":"failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint \\"inv_caboose_pkey\\""}
73122025-04-30T06:42:59.389ZDEBGtest_snapshot_basic (BackgroundTasks): activating background_task = inventory_collection iteration = 23 reason = Signaled
73132025-04-30T06:42:59.389ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
73142025-04-30T06:42:59.389ZTRACtest_snapshot_basic (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
73152025-04-30T06:42:59.389ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
73162025-04-30T06:42:59.389ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
73172025-04-30T06:42:59.389ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
73182025-04-30T06:42:59.391ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
73192025-04-30T06:42:59.397ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
73202025-04-30T06:42:59.397ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
73212025-04-30T06:42:59.397ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
73222025-04-30T06:42:59.397ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
73232025-04-30T06:42:59.398ZDEBGtest_snapshot_basic: attempt to fetch latest inventory collection result = Ok("found")
73242025-04-30T06:42:59.404ZDEBGtest_snapshot_basic (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = [(d1ad44e5-18dd-4193-a70e-bf9042f0e0e1, 0)]
73252025-04-30T06:42:59.404ZTRACtest_snapshot_basic (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
73262025-04-30T06:42:59.405ZTRACtest_snapshot_basic (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: 81 3a 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 .:..........._mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:54608 req_id = 9acc57aa-1585-43c9-9c81-414d69244aaa
73272025-04-30T06:42:59.406ZDEBGtest_snapshot_basic (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 33082,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:54608 req_id = 9acc57aa-1585-43c9-9c81-414d69244aaa
73282025-04-30T06:42:59.406ZDEBGtest_snapshot_basic (store): query key key = _mgs._tcp
73292025-04-30T06:42:59.406ZDEBGtest_snapshot_basic (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
73302025-04-30T06:42:59.406ZDEBGtest_snapshot_basic (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:54608 query = LowerQuery { name: LowerName(Name("_mgs._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 41502, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = 9acc57aa-1585-43c9-9c81-414d69244aaa
73312025-04-30T06:42:59.407ZDEBGtest_snapshot_basic (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 41502, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 4244, tv_nsec: 206236368 } })
73322025-04-30T06:42:59.407ZTRACtest_snapshot_basic (DnsResolver): lookup_service_targets: looking up SRV target
73332025-04-30T06:42:59.407ZTRACtest_snapshot_basic (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: 0d 59 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e .Y..........-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:51214 req_id = a5fa0beb-7b3f-4eeb-8e6b-0b550cd172dc
73342025-04-30T06:42:59.407ZDEBGtest_snapshot_basic (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 3417,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:51214 req_id = a5fa0beb-7b3f-4eeb-8e6b-0b550cd172dc
73352025-04-30T06:42:59.407ZDEBGtest_snapshot_basic (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
73362025-04-30T06:42:59.407ZDEBGtest_snapshot_basic (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:51214 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = a5fa0beb-7b3f-4eeb-8e6b-0b550cd172dc
73372025-04-30T06:42:59.494ZTRACtest_snapshot_basic (DnsResolver): lookup_all_socket_v6 srv dns_name = _clickhouse-admin-keeper._tcp.control-plane.oxide.internal
73382025-04-30T06:42:59.494ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
73392025-04-30T06:42:59.494ZTRACtest_snapshot_basic (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
73402025-04-30T06:42:59.494ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
73412025-04-30T06:42:59.494ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {} }
73422025-04-30T06:42:59.495ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
73432025-04-30T06:42:59.496ZTRACtest_snapshot_basic (dns): buffer buffer = Length: 87 (0x57) bytes\n0000: 9d 67 01 00 00 01 00 00 00 00 00 01 18 5f 63 6c .g..........._cl\n0010: 69 63 6b 68 6f 75 73 65 2d 61 64 6d 69 6e 2d 6b ickhouse-admin-k\n0020: 65 65 70 65 72 04 5f 74 63 70 0d 63 6f 6e 74 72 eeper._tcp.contr\n0030: 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 69 ol-plane.oxide.i\n0040: 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 04 nternal..!....).\n0050: d0 00 00 00 00 00 00 ....... kind = dns peer_addr = [::1]:52974 req_id = 82e890ec-4f21-491d-9632-8f32e5e55381
73442025-04-30T06:42:59.496ZDEBGtest_snapshot_basic (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 40295,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 24,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 97,\n 100,\n 109,\n 105,\n 110,\n 45,\n 107,\n 101,\n 101,\n 112,\n 101,\n 114,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:52974 req_id = 82e890ec-4f21-491d-9632-8f32e5e55381
73452025-04-30T06:42:59.497ZDEBGtest_snapshot_basic (store): query key key = _clickhouse-admin-keeper._tcp
73462025-04-30T06:42:59.497ZERROtest_snapshot_basic (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 40295, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } NXDOMAIN: no records found for name: "_clickhouse-admin-keeper._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:52974 req_id = 82e890ec-4f21-491d-9632-8f32e5e55381
73472025-04-30T06:42:59.497ZDEBGtest_snapshot_basic (BackgroundTasks): begin collection background_task = inventory_collection
73482025-04-30T06:42:59.497ZDEBGtest_snapshot_basic (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:41502
73492025-04-30T06:42:59.497ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/ignition
73502025-04-30T06:42:59.499ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
73512025-04-30T06:42:59.504ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
73522025-04-30T06:42:59.504ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
73532025-04-30T06:42:59.504ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {} }
73542025-04-30T06:42:59.504ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
73552025-04-30T06:42:59.507ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "f40a2979-db60-424d-a9ea-bfa5de618843", "content-length": "835", "date": "Wed, 30 Apr 2025 06:42:58 GMT"} })
73562025-04-30T06:42:59.507ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/0
73572025-04-30T06:42:59.507ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
73582025-04-30T06:42:59.507ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {} }
73592025-04-30T06:42:59.508ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
73602025-04-30T06:42:59.521ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "bf3e79bf-163b-4db0-acef-4d2a8d62a50c", "content-length": "734", "date": "Wed, 30 Apr 2025 06:42:58 GMT"} })
73612025-04-30T06:42:59.531ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/0/component/sp/caboose?firmware_slot=0
73622025-04-30T06:42:59.558ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "736aee1d-8c14-49ae-9fc3-296df0e413f3", "content-length": "111", "date": "Wed, 30 Apr 2025 06:42:59 GMT"} })
73632025-04-30T06:42:59.558ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/0/component/sp/caboose?firmware_slot=1
73642025-04-30T06:42:59.562ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "00a9cf21-81fe-448d-8837-a8f60ab85cef", "content-length": "111", "date": "Wed, 30 Apr 2025 06:42:59 GMT"} })
73652025-04-30T06:42:59.563ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/0/component/rot/caboose?firmware_slot=0
73662025-04-30T06:42:59.571ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "508532a4-f356-4eff-a442-3c3c090422aa", "content-length": "170", "date": "Wed, 30 Apr 2025 06:42:59 GMT"} })
73672025-04-30T06:42:59.571ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/0/component/rot/caboose?firmware_slot=1
73682025-04-30T06:42:59.577ZDEBGtest_snapshot_basic: attempt to fetch latest inventory collection result = Ok("found")
73692025-04-30T06:42:59.577ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "8fe051fe-74ca-425c-9a14-cb78cd53d198", "content-length": "170", "date": "Wed, 30 Apr 2025 06:42:59 GMT"} })
73702025-04-30T06:42:59.577ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/0/component/stage0/caboose?firmware_slot=0
73712025-04-30T06:42:59.581ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "1921659e-1eb9-4a0a-a9c3-204b22ee7802", "content-length": "179", "date": "Wed, 30 Apr 2025 06:42:59 GMT"} })
73722025-04-30T06:42:59.582ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/0/component/stage0/caboose?firmware_slot=1
73732025-04-30T06:42:59.586ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "42bf318e-776c-4bb6-9277-291a344de4b6", "content-length": "179", "date": "Wed, 30 Apr 2025 06:42:59 GMT"} })
73742025-04-30T06:42:59.586ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/0/component/rot/cmpa
73752025-04-30T06:42:59.588ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "dbd42eb5-d714-4945-aa4b-8cd68490bcb1", "content-length": "702", "date": "Wed, 30 Apr 2025 06:42:59 GMT"} })
73762025-04-30T06:42:59.588ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/0/component/rot/cfpa
73772025-04-30T06:42:59.589ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "3b5bfdb1-7edb-4aba-affd-dfb7eb3006de", "content-length": "727", "date": "Wed, 30 Apr 2025 06:42:59 GMT"} })
73782025-04-30T06:42:59.589ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/0/component/rot/cfpa
73792025-04-30T06:42:59.591ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "782e9d81-e5c3-460e-867e-6d1983334f05", "content-length": "729", "date": "Wed, 30 Apr 2025 06:42:59 GMT"} })
73802025-04-30T06:42:59.591ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/0/component/rot/cfpa
73812025-04-30T06:42:59.594ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "8c2e588d-4639-46bf-81cc-f7d1ea051371", "content-length": "728", "date": "Wed, 30 Apr 2025 06:42:59 GMT"} })
73822025-04-30T06:42:59.594ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/1
73832025-04-30T06:42:59.596ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "66d93a8a-91eb-4773-a390-19464d5c7432", "content-length": "734", "date": "Wed, 30 Apr 2025 06:42:59 GMT"} })
73842025-04-30T06:42:59.596ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/1/component/sp/caboose?firmware_slot=0
73852025-04-30T06:42:59.599ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "d90e887b-e6da-49fb-b44b-08f3f749cc60", "content-length": "111", "date": "Wed, 30 Apr 2025 06:42:59 GMT"} })
73862025-04-30T06:42:59.599ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/1/component/sp/caboose?firmware_slot=1
73872025-04-30T06:42:59.604ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "28c89ba2-5adf-4ece-8bf3-93b320b4da50", "content-length": "111", "date": "Wed, 30 Apr 2025 06:42:59 GMT"} })
73882025-04-30T06:42:59.605ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/1/component/rot/caboose?firmware_slot=0
73892025-04-30T06:42:59.609ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "c045984c-5334-42ec-bff9-d6aad6174d83", "content-length": "170", "date": "Wed, 30 Apr 2025 06:42:59 GMT"} })
73902025-04-30T06:42:59.609ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/1/component/rot/caboose?firmware_slot=1
73912025-04-30T06:42:59.611ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "fce2d9f4-11a6-4164-9c25-74832ab8c8f2", "content-length": "170", "date": "Wed, 30 Apr 2025 06:42:59 GMT"} })
73922025-04-30T06:42:59.611ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/1/component/stage0/caboose?firmware_slot=0
73932025-04-30T06:42:59.613ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "75afad76-f33c-4d3a-93c4-250049e5712b", "content-length": "179", "date": "Wed, 30 Apr 2025 06:42:59 GMT"} })
73942025-04-30T06:42:59.613ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/1/component/stage0/caboose?firmware_slot=1
73952025-04-30T06:42:59.617ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "c7d3ef72-2139-4ad8-9a74-7318310a0d7d", "content-length": "179", "date": "Wed, 30 Apr 2025 06:42:59 GMT"} })
73962025-04-30T06:42:59.621ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/1/component/rot/cmpa
73972025-04-30T06:42:59.624ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "41d85d05-85d0-4a96-bf58-65886e589210", "content-length": "702", "date": "Wed, 30 Apr 2025 06:42:59 GMT"} })
73982025-04-30T06:42:59.625ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/1/component/rot/cfpa
73992025-04-30T06:42:59.627ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "3ba7e295-f44d-4b52-901f-e8a57b98245b", "content-length": "727", "date": "Wed, 30 Apr 2025 06:42:59 GMT"} })
74002025-04-30T06:42:59.627ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/1/component/rot/cfpa
74012025-04-30T06:42:59.627ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
74022025-04-30T06:42:59.627ZTRACtest_snapshot_basic (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
74032025-04-30T06:42:59.627ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
74042025-04-30T06:42:59.627ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {} }
74052025-04-30T06:42:59.629ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
74062025-04-30T06:42:59.633ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "33fad8a5-edce-4d97-9407-b9adacb58a6b", "content-length": "729", "date": "Wed, 30 Apr 2025 06:42:59 GMT"} })
74072025-04-30T06:42:59.634ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/1/component/rot/cfpa
74082025-04-30T06:42:59.641ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
74092025-04-30T06:42:59.658ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
74102025-04-30T06:42:59.658ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
74112025-04-30T06:42:59.659ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {} }
74122025-04-30T06:42:59.661ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
74132025-04-30T06:42:59.662ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "89a2f721-c9bc-435a-b5f6-e59453e049cc", "content-length": "728", "date": "Wed, 30 Apr 2025 06:42:59 GMT"} })
74142025-04-30T06:42:59.663ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/0
74152025-04-30T06:42:59.666ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
74162025-04-30T06:42:59.666ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {} }
74172025-04-30T06:42:59.667ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
74182025-04-30T06:42:59.671ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "fe899f00-d38b-48b0-827d-b5723aede7d5", "content-length": "723", "date": "Wed, 30 Apr 2025 06:42:59 GMT"} })
74192025-04-30T06:42:59.671ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/0/component/sp/caboose?firmware_slot=0
74202025-04-30T06:42:59.678ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "d03df236-f40d-404d-887a-b389edf3df49", "content-length": "109", "date": "Wed, 30 Apr 2025 06:42:59 GMT"} })
74212025-04-30T06:42:59.678ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/0/component/sp/caboose?firmware_slot=1
74222025-04-30T06:42:59.682ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "e6942ffc-e44d-4824-bfe9-7a271499f538", "content-length": "109", "date": "Wed, 30 Apr 2025 06:42:59 GMT"} })
74232025-04-30T06:42:59.682ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/0/component/rot/caboose?firmware_slot=0
74242025-04-30T06:42:59.686ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "298f849a-8988-4d3f-8c43-0bb5988afba8", "content-length": "169", "date": "Wed, 30 Apr 2025 06:42:59 GMT"} })
74252025-04-30T06:42:59.686ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/0/component/rot/caboose?firmware_slot=1
74262025-04-30T06:42:59.689ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "45cd77ec-9b87-4a6c-803c-0819a0622a54", "content-length": "169", "date": "Wed, 30 Apr 2025 06:42:59 GMT"} })
74272025-04-30T06:42:59.690ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/0/component/stage0/caboose?firmware_slot=0
74282025-04-30T06:42:59.692ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "f8307cc2-c559-407b-8d9b-5189a21ad1db", "content-length": "178", "date": "Wed, 30 Apr 2025 06:42:59 GMT"} })
74292025-04-30T06:42:59.699ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/0/component/stage0/caboose?firmware_slot=1
74302025-04-30T06:42:59.704ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "614f74c0-65b2-4e29-8383-b20069380763", "content-length": "178", "date": "Wed, 30 Apr 2025 06:42:59 GMT"} })
74312025-04-30T06:42:59.705ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/0/component/rot/cmpa
74322025-04-30T06:42:59.705ZDEBGtest_snapshot_basic: attempt to fetch latest inventory collection result = Ok("found")
74332025-04-30T06:42:59.707ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "1c41a1ed-6d95-43bb-a92e-a904b096b7af", "content-length": "702", "date": "Wed, 30 Apr 2025 06:42:59 GMT"} })
74342025-04-30T06:42:59.707ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/0/component/rot/cfpa
74352025-04-30T06:42:59.709ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "2acf31ab-bb23-46e9-95fc-7c995e5ae067", "content-length": "727", "date": "Wed, 30 Apr 2025 06:42:59 GMT"} })
74362025-04-30T06:42:59.709ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/0/component/rot/cfpa
74372025-04-30T06:42:59.711ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "3af18266-4845-4284-bcfe-6e79609035f9", "content-length": "729", "date": "Wed, 30 Apr 2025 06:42:59 GMT"} })
74382025-04-30T06:42:59.711ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/0/component/rot/cfpa
74392025-04-30T06:42:59.712ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "5e9b88f0-05b1-44c0-8f87-50399f2d61eb", "content-length": "728", "date": "Wed, 30 Apr 2025 06:42:59 GMT"} })
74402025-04-30T06:42:59.712ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/1
74412025-04-30T06:42:59.714ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "b33ef74f-4e85-424c-b6f5-4a334e2006a3", "content-length": "723", "date": "Wed, 30 Apr 2025 06:42:59 GMT"} })
74422025-04-30T06:42:59.715ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/1/component/sp/caboose?firmware_slot=0
74432025-04-30T06:42:59.717ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "1e9e43bb-f22e-4a38-b377-5a645db5a397", "content-length": "109", "date": "Wed, 30 Apr 2025 06:42:59 GMT"} })
74442025-04-30T06:42:59.717ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/1/component/sp/caboose?firmware_slot=1
74452025-04-30T06:42:59.719ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "c2388ef0-266e-43c7-9c52-b09783d92876", "content-length": "109", "date": "Wed, 30 Apr 2025 06:42:59 GMT"} })
74462025-04-30T06:42:59.720ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/1/component/rot/caboose?firmware_slot=0
74472025-04-30T06:42:59.720ZINFOtest_snapshot_basic (oximeter-agent): refreshing list of producers from Nexus collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 file = oximeter/collector/src/agent.rs:562
74482025-04-30T06:42:59.721ZDEBGtest_snapshot_basic (oximeter): client request body = None method = GET uri = http://127.0.0.1:61468/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
74492025-04-30T06:42:59.721ZTRACtest_snapshot_basic (dropshot_internal): incoming request local_addr = 127.0.0.1:61468 method = GET remote_addr = 127.0.0.1:64332 req_id = 5a5520b7-b03e-47f0-9b38-0f4702f9b5ce uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
74502025-04-30T06:42:59.722ZTRACtest_snapshot_basic (dropshot_internal): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:61468 method = GET remote_addr = 127.0.0.1:64332 req_id = 5a5520b7-b03e-47f0-9b38-0f4702f9b5ce resource = Database uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
74512025-04-30T06:42:59.722ZDEBGtest_snapshot_basic (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:61468 method = GET remote_addr = 127.0.0.1:64332 req_id = 5a5520b7-b03e-47f0-9b38-0f4702f9b5ce roles = RoleSet { roles: {} } uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
74522025-04-30T06:42:59.723ZDEBGtest_snapshot_basic (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:61468 method = GET remote_addr = 127.0.0.1:64332 req_id = 5a5520b7-b03e-47f0-9b38-0f4702f9b5ce resource = Database result = Ok(()) uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
74532025-04-30T06:42:59.725ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "28e398ac-8809-435a-bdc7-aaa3e0805ea8", "content-length": "169", "date": "Wed, 30 Apr 2025 06:42:59 GMT"} })
74542025-04-30T06:42:59.726ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/1/component/rot/caboose?firmware_slot=1
74552025-04-30T06:42:59.726ZINFOtest_snapshot_basic (dropshot_internal): request completed latency_us = 5223 local_addr = 127.0.0.1:61468 method = GET remote_addr = 127.0.0.1:64332 req_id = 5a5520b7-b03e-47f0-9b38-0f4702f9b5ce response_code = 200 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
74562025-04-30T06:42:59.727ZDEBGtest_snapshot_basic (oximeter): client response result = Ok(Response { url: "http://127.0.0.1:61468/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending", status: 200, headers: {"content-type": "application/json", "x-request-id": "5a5520b7-b03e-47f0-9b38-0f4702f9b5ce", "content-length": "405", "date": "Wed, 30 Apr 2025 06:42:59 GMT"} })
74572025-04-30T06:42:59.728ZDEBGtest_snapshot_basic (oximeter): client request body = None method = GET uri = http://127.0.0.1:61468/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
74582025-04-30T06:42:59.729ZTRACtest_snapshot_basic (dropshot_internal): incoming request local_addr = 127.0.0.1:61468 method = GET remote_addr = 127.0.0.1:64332 req_id = a516d881-982a-4701-9cb0-b604439ea7d3 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
74592025-04-30T06:42:59.729ZTRACtest_snapshot_basic (dropshot_internal): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:61468 method = GET remote_addr = 127.0.0.1:64332 req_id = a516d881-982a-4701-9cb0-b604439ea7d3 resource = Database uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
74602025-04-30T06:42:59.729ZDEBGtest_snapshot_basic (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:61468 method = GET remote_addr = 127.0.0.1:64332 req_id = a516d881-982a-4701-9cb0-b604439ea7d3 roles = RoleSet { roles: {} } uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
74612025-04-30T06:42:59.730ZDEBGtest_snapshot_basic (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:61468 method = GET remote_addr = 127.0.0.1:64332 req_id = a516d881-982a-4701-9cb0-b604439ea7d3 resource = Database result = Ok(()) uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
74622025-04-30T06:42:59.733ZINFOtest_snapshot_basic (dropshot_internal): request completed latency_us = 4074 local_addr = 127.0.0.1:61468 method = GET remote_addr = 127.0.0.1:64332 req_id = a516d881-982a-4701-9cb0-b604439ea7d3 response_code = 200 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
74632025-04-30T06:42:59.733ZDEBGtest_snapshot_basic (oximeter): client response result = Ok(Response { url: "http://127.0.0.1:61468/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D", status: 200, headers: {"content-type": "application/json", "x-request-id": "a516d881-982a-4701-9cb0-b604439ea7d3", "content-length": "29", "date": "Wed, 30 Apr 2025 06:42:59 GMT"} })
74642025-04-30T06:42:59.733ZTRACtest_snapshot_basic (oximeter-agent): ignoring request to update existing metric producer, since the endpoint information is the same as the existing collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = 354a52d4-3078-4706-81ce-12c32f6bbeaa
74652025-04-30T06:42:59.733ZTRACtest_snapshot_basic (oximeter-agent): ignoring request to update existing metric producer, since the endpoint information is the same as the existing collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
74662025-04-30T06:42:59.733ZINFOtest_snapshot_basic (oximeter-agent): refreshed list of producers from Nexus collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 file = oximeter/collector/src/agent.rs:637 n_current_tasks = 2 n_pruned_tasks = 0
74672025-04-30T06:42:59.737ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "96b4e327-9d00-4da7-8d37-e331b754b53d", "content-length": "169", "date": "Wed, 30 Apr 2025 06:42:59 GMT"} })
74682025-04-30T06:42:59.737ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/1/component/stage0/caboose?firmware_slot=0
74692025-04-30T06:42:59.739ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "c1c57e5e-5e55-47ab-bbd3-d4ce3dbb6470", "content-length": "178", "date": "Wed, 30 Apr 2025 06:42:59 GMT"} })
74702025-04-30T06:42:59.740ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/1/component/stage0/caboose?firmware_slot=1
74712025-04-30T06:42:59.741ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "6a9cced5-f3b7-472c-a014-25948191aa91", "content-length": "178", "date": "Wed, 30 Apr 2025 06:42:59 GMT"} })
74722025-04-30T06:42:59.742ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/1/component/rot/cmpa
74732025-04-30T06:42:59.743ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "a930cc63-8d75-4a51-b3e0-8a8844c75105", "content-length": "702", "date": "Wed, 30 Apr 2025 06:42:59 GMT"} })
74742025-04-30T06:42:59.743ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/1/component/rot/cfpa
74752025-04-30T06:42:59.743ZDEBGtest_snapshot_basic (collection-task): sent timer-based collection request to the collection task collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = 354a52d4-3078-4706-81ce-12c32f6bbeaa
74762025-04-30T06:42:59.743ZDEBGtest_snapshot_basic (collection-task): collecting from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = 354a52d4-3078-4706-81ce-12c32f6bbeaa
74772025-04-30T06:42:59.744ZINFOtest_snapshot_basic (dropshot): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:1025 local_addr = 127.0.0.1:47319 remote_addr = 127.0.0.1:51756
74782025-04-30T06:42:59.744ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "5f63625b-df7d-42c1-bb3b-a7df6f5b3cd6", "content-length": "727", "date": "Wed, 30 Apr 2025 06:42:59 GMT"} })
74792025-04-30T06:42:59.744ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/1/component/rot/cfpa
74802025-04-30T06:42:59.745ZTRACtest_snapshot_basic (dropshot): incoming request local_addr = 127.0.0.1:47319 method = GET remote_addr = 127.0.0.1:51756 req_id = 97d6e5cd-00e8-44b4-86dc-dfc15112c88f uri = /354a52d4-3078-4706-81ce-12c32f6bbeaa
74812025-04-30T06:42:59.749ZINFOtest_snapshot_basic (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 4562 local_addr = 127.0.0.1:47319 method = GET remote_addr = 127.0.0.1:51756 req_id = 97d6e5cd-00e8-44b4-86dc-dfc15112c88f response_code = 200 uri = /354a52d4-3078-4706-81ce-12c32f6bbeaa
74822025-04-30T06:42:59.750ZTRACtest_snapshot_basic (collection-task): sent collection request to producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = 354a52d4-3078-4706-81ce-12c32f6bbeaa
74832025-04-30T06:42:59.774ZDEBGtest_snapshot_basic (collection-task): collected results from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 n_results = 5 producer_id = 354a52d4-3078-4706-81ce-12c32f6bbeaa
74842025-04-30T06:42:59.775ZTRACtest_snapshot_basic (collection-task): forwarded results to main collection loop collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = 354a52d4-3078-4706-81ce-12c32f6bbeaa
74852025-04-30T06:42:59.775ZTRACtest_snapshot_basic (collection-task): top of inner collection loop, waiting for next request collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = 354a52d4-3078-4706-81ce-12c32f6bbeaa
74862025-04-30T06:42:59.775ZDEBGtest_snapshot_basic (results-sink): inserting 17 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
74872025-04-30T06:42:59.776ZDEBGtest_snapshot_basic (results-sink-cluster): inserting 17 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
74882025-04-30T06:42:59.776ZDEBGtest_snapshot_basic (clickhouse-client): unrolling 17 total samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 6ad06c6b-9a6e-4ed1-b3f4-5d2e9da3f8a7
74892025-04-30T06:42:59.782ZDEBGtest_snapshot_basic (clickhouse-client): retrieving timeseries schema from database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 6ad06c6b-9a6e-4ed1-b3f4-5d2e9da3f8a7
74902025-04-30T06:42:59.782ZTRACtest_snapshot_basic (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 6ad06c6b-9a6e-4ed1-b3f4-5d2e9da3f8a7 sql = SELECT * FROM oximeter.timeseries_schema WHERE timeseries_name NOT IN ('integration_target:integration_metric') FORMAT Native;
74912025-04-30T06:42:59.784ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
74922025-04-30T06:42:59.784ZTRACtest_snapshot_basic (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
74932025-04-30T06:42:59.785ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
74942025-04-30T06:42:59.785ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {} }
74952025-04-30T06:42:59.785ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
74962025-04-30T06:42:59.789ZTRACtest_snapshot_basic (clickhouse-client): no new timeseries schema in database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 6ad06c6b-9a6e-4ed1-b3f4-5d2e9da3f8a7
74972025-04-30T06:42:59.790ZDEBGtest_snapshot_basic (clickhouse-client): new timeseries schema collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 6ad06c6b-9a6e-4ed1-b3f4-5d2e9da3f8a7 schema = TimeseriesSchema { timeseries_name: TimeseriesName("http_service:request_latency_histogram"), description: TimeseriesDescription { target: "", metric: "" }, field_schema: {FieldSchema { name: "id", field_type: Uuid, source: Target, description: "" }, FieldSchema { name: "name", field_type: String, source: Target, description: "" }, FieldSchema { name: "operation_id", field_type: String, source: Metric, description: "" }, FieldSchema { name: "status_code", field_type: U16, source: Metric, description: "" }}, datum_type: HistogramU64, version: 1, authz_scope: Fleet, units: Count, created: 2025-04-30T06:42:59.782405934Z } timeseries_name = http_service:request_latency_histogram
74982025-04-30T06:42:59.796ZDEBGtest_snapshot_basic (clickhouse-client): retrieving timeseries schema from database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 6ad06c6b-9a6e-4ed1-b3f4-5d2e9da3f8a7
74992025-04-30T06:42:59.796ZTRACtest_snapshot_basic (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 6ad06c6b-9a6e-4ed1-b3f4-5d2e9da3f8a7 sql = SELECT * FROM oximeter.timeseries_schema WHERE timeseries_name NOT IN ('http_service:request_latency_histogram', 'integration_target:integration_metric') FORMAT Native;
75002025-04-30T06:42:59.798ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "62a9dd6f-b171-455c-882e-bc205244bb46", "content-length": "729", "date": "Wed, 30 Apr 2025 06:42:59 GMT"} })
75012025-04-30T06:42:59.800ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/1/component/rot/cfpa
75022025-04-30T06:42:59.802ZDEBGtest_snapshot_basic (collection-task): sent timer-based collection request to the collection task collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
75032025-04-30T06:42:59.802ZDEBGtest_snapshot_basic (collection-task): collecting from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
75042025-04-30T06:42:59.803ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
75052025-04-30T06:42:59.815ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
75062025-04-30T06:42:59.815ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
75072025-04-30T06:42:59.815ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {} }
75082025-04-30T06:42:59.816ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
75092025-04-30T06:42:59.817ZTRACtest_snapshot_basic (clickhouse-client): no new timeseries schema in database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 6ad06c6b-9a6e-4ed1-b3f4-5d2e9da3f8a7
75102025-04-30T06:42:59.817ZDEBGtest_snapshot_basic (clickhouse-client): new timeseries schema collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 6ad06c6b-9a6e-4ed1-b3f4-5d2e9da3f8a7 schema = TimeseriesSchema { timeseries_name: TimeseriesName("collection_target:cpus_provisioned"), description: TimeseriesDescription { target: "", metric: "" }, field_schema: {FieldSchema { name: "id", field_type: Uuid, source: Target, description: "" }}, datum_type: I64, version: 1, authz_scope: Fleet, units: Count, created: 2025-04-30T06:42:59.796663130Z } timeseries_name = collection_target:cpus_provisioned
75112025-04-30T06:42:59.817ZDEBGtest_snapshot_basic (clickhouse-client): retrieving timeseries schema from database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 6ad06c6b-9a6e-4ed1-b3f4-5d2e9da3f8a7
75122025-04-30T06:42:59.817ZTRACtest_snapshot_basic (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 6ad06c6b-9a6e-4ed1-b3f4-5d2e9da3f8a7 sql = SELECT * FROM oximeter.timeseries_schema WHERE timeseries_name NOT IN ('collection_target:cpus_provisioned', 'http_service:request_latency_histogram', 'integration_target:integration_metric') FORMAT Native;
75132025-04-30T06:42:59.822ZTRACtest_snapshot_basic (clickhouse-client): no new timeseries schema in database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 6ad06c6b-9a6e-4ed1-b3f4-5d2e9da3f8a7
75142025-04-30T06:42:59.822ZDEBGtest_snapshot_basic (clickhouse-client): new timeseries schema collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 6ad06c6b-9a6e-4ed1-b3f4-5d2e9da3f8a7 schema = TimeseriesSchema { timeseries_name: TimeseriesName("collection_target:ram_provisioned"), description: TimeseriesDescription { target: "", metric: "" }, field_schema: {FieldSchema { name: "id", field_type: Uuid, source: Target, description: "" }}, datum_type: I64, version: 1, authz_scope: Fleet, units: Count, created: 2025-04-30T06:42:59.817850204Z } timeseries_name = collection_target:ram_provisioned
75152025-04-30T06:42:59.823ZDEBGtest_snapshot_basic (clickhouse-client): retrieving timeseries schema from database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 6ad06c6b-9a6e-4ed1-b3f4-5d2e9da3f8a7
75162025-04-30T06:42:59.827ZTRACtest_snapshot_basic (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 6ad06c6b-9a6e-4ed1-b3f4-5d2e9da3f8a7 sql = SELECT * FROM oximeter.timeseries_schema WHERE timeseries_name NOT IN ('collection_target:cpus_provisioned', 'collection_target:ram_provisioned', 'http_service:request_latency_histogram', 'integration_target:integration_metric') FORMAT Native;
75172025-04-30T06:42:59.829ZTRACtest_snapshot_basic (collection-task): sent collection request to producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
75182025-04-30T06:42:59.830ZDEBGtest_snapshot_basic (collection-task): collected results from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 n_results = 1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
75192025-04-30T06:42:59.830ZTRACtest_snapshot_basic (collection-task): forwarded results to main collection loop collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
75202025-04-30T06:42:59.830ZTRACtest_snapshot_basic (collection-task): top of inner collection loop, waiting for next request collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
75212025-04-30T06:42:59.832ZTRACtest_snapshot_basic (clickhouse-client): no new timeseries schema in database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 6ad06c6b-9a6e-4ed1-b3f4-5d2e9da3f8a7
75222025-04-30T06:42:59.832ZDEBGtest_snapshot_basic (clickhouse-client): new timeseries schema collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 6ad06c6b-9a6e-4ed1-b3f4-5d2e9da3f8a7 schema = TimeseriesSchema { timeseries_name: TimeseriesName("collection_target:virtual_disk_space_provisioned"), description: TimeseriesDescription { target: "", metric: "" }, field_schema: {FieldSchema { name: "id", field_type: Uuid, source: Target, description: "" }}, datum_type: I64, version: 1, authz_scope: Fleet, units: Count, created: 2025-04-30T06:42:59.823169923Z } timeseries_name = collection_target:virtual_disk_space_provisioned
75232025-04-30T06:42:59.843ZDEBGtest_snapshot_basic (clickhouse-client): inserting 4 new timeseries schema collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 6ad06c6b-9a6e-4ed1-b3f4-5d2e9da3f8a7
75242025-04-30T06:42:59.843ZTRACtest_snapshot_basic (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 6ad06c6b-9a6e-4ed1-b3f4-5d2e9da3f8a7 n_columns = 6 n_rows = 4 sql = INSERT INTO oximeter.timeseries_schema FORMAT Native
75252025-04-30T06:42:59.843ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "87b74d3b-bf0f-4b2d-b3aa-cfc416b04ee1", "content-length": "728", "date": "Wed, 30 Apr 2025 06:42:59 GMT"} })
75262025-04-30T06:42:59.843ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
75272025-04-30T06:42:59.843ZTRACtest_snapshot_basic (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
75282025-04-30T06:42:59.844ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
75292025-04-30T06:42:59.844ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
75302025-04-30T06:42:59.844ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
75312025-04-30T06:42:59.845ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
75322025-04-30T06:42:59.845ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {} }
75332025-04-30T06:42:59.845ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
75342025-04-30T06:42:59.850ZTRACtest_snapshot_basic (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 6ad06c6b-9a6e-4ed1-b3f4-5d2e9da3f8a7 n_columns = 4 n_rows = 10 sql = INSERT INTO oximeter.fields_string FORMAT Native
75352025-04-30T06:42:59.851ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
75362025-04-30T06:42:59.855ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
75372025-04-30T06:42:59.855ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
75382025-04-30T06:42:59.855ZTRACtest_snapshot_basic (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
75392025-04-30T06:42:59.855ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
75402025-04-30T06:42:59.855ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
75412025-04-30T06:42:59.856ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
75422025-04-30T06:42:59.861ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
75432025-04-30T06:42:59.865ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
75442025-04-30T06:42:59.867ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
75452025-04-30T06:42:59.867ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
75462025-04-30T06:42:59.867ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
75472025-04-30T06:42:59.869ZDEBGtest_snapshot_basic (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 6ad06c6b-9a6e-4ed1-b3f4-5d2e9da3f8a7 n_rows = 10 table_name = fields_string
75482025-04-30T06:42:59.869ZTRACtest_snapshot_basic (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 6ad06c6b-9a6e-4ed1-b3f4-5d2e9da3f8a7 n_columns = 4 n_rows = 5 sql = INSERT INTO oximeter.fields_u16 FORMAT Native
75492025-04-30T06:42:59.934ZDEBGtest_snapshot_basic (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:41923
75502025-04-30T06:42:59.934ZDEBGtest_snapshot_basic (BackgroundTasks): client request SledAgent = http://[::1]:41923 background_task = inventory_collection body = None method = GET uri = http://[::1]:41923/inventory
75512025-04-30T06:42:59.935ZINFOtest_snapshot_basic (dropshot): accepted connection local_addr = [::1]:41923 remote_addr = [::1]:56426 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
75522025-04-30T06:42:59.936ZDEBGtest_snapshot_basic (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 6ad06c6b-9a6e-4ed1-b3f4-5d2e9da3f8a7 n_rows = 5 table_name = fields_u16
75532025-04-30T06:42:59.936ZTRACtest_snapshot_basic (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 6ad06c6b-9a6e-4ed1-b3f4-5d2e9da3f8a7 n_columns = 4 n_rows = 17 sql = INSERT INTO oximeter.fields_uuid FORMAT Native
75542025-04-30T06:42:59.936ZWARNtest_snapshot_basic (results-sink-cluster): failed to insert some results into metric DB collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 error = Failed to check out connection to database: No backends found for this service file = oximeter/collector/src/results_sink.rs:92
75552025-04-30T06:42:59.939ZTRACtest_snapshot_basic (dropshot): incoming request local_addr = [::1]:41923 method = GET remote_addr = [::1]:56426 req_id = 99fa5e90-0f6b-40b3-a649-bd72c31eac4d sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
75562025-04-30T06:42:59.939ZINFOtest_snapshot_basic (dropshot): request completed latency_us = 677 local_addr = [::1]:41923 method = GET remote_addr = [::1]:56426 req_id = 99fa5e90-0f6b-40b3-a649-bd72c31eac4d response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
75572025-04-30T06:42:59.940ZDEBGtest_snapshot_basic (BackgroundTasks): client response SledAgent = http://[::1]:41923 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:41923/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "99fa5e90-0f6b-40b3-a649-bd72c31eac4d", "content-length": "3141", "date": "Wed, 30 Apr 2025 06:42:59 GMT"} })
75582025-04-30T06:42:59.943ZDEBGtest_snapshot_basic (BackgroundTasks): begin collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
75592025-04-30T06:42:59.943ZDEBGtest_snapshot_basic (BackgroundTasks): end collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
75602025-04-30T06:42:59.943ZDEBGtest_snapshot_basic (BackgroundTasks): finished collection background_task = inventory_collection
75612025-04-30T06:42:59.946ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
75622025-04-30T06:42:59.946ZTRACtest_snapshot_basic (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
75632025-04-30T06:42:59.946ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
75642025-04-30T06:42:59.946ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
75652025-04-30T06:42:59.947ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
75662025-04-30T06:42:59.950ZDEBGtest_snapshot_basic (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 6ad06c6b-9a6e-4ed1-b3f4-5d2e9da3f8a7 n_rows = 17 table_name = fields_uuid
75672025-04-30T06:42:59.951ZTRACtest_snapshot_basic (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 6ad06c6b-9a6e-4ed1-b3f4-5d2e9da3f8a7 n_columns = 19 n_rows = 5 sql = INSERT INTO oximeter.measurements_histogramu64 FORMAT Native
75682025-04-30T06:42:59.953ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
75692025-04-30T06:42:59.969ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
75702025-04-30T06:42:59.969ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
75712025-04-30T06:42:59.969ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
75722025-04-30T06:42:59.969ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
75732025-04-30T06:42:59.973ZDEBGtest_snapshot_basic (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 6ad06c6b-9a6e-4ed1-b3f4-5d2e9da3f8a7 n_rows = 5 table_name = measurements_histogramu64
75742025-04-30T06:42:59.973ZTRACtest_snapshot_basic (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 6ad06c6b-9a6e-4ed1-b3f4-5d2e9da3f8a7 n_columns = 4 n_rows = 12 sql = INSERT INTO oximeter.measurements_i64 FORMAT Native
75752025-04-30T06:42:59.981ZDEBGtest_snapshot_basic (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 6ad06c6b-9a6e-4ed1-b3f4-5d2e9da3f8a7 n_rows = 12 table_name = measurements_i64
75762025-04-30T06:42:59.981ZTRACtest_snapshot_basic (results-sink): successfully inserted samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
75772025-04-30T06:42:59.997ZDEBGtest_snapshot_basic: attempt to fetch latest inventory collection result = Ok("found")
75782025-04-30T06:43:00.023ZWARNtest_snapshot_basic (datastore): Non-retryable transaction failure err = DatabaseError(UniqueViolation, "duplicate key value violates unique constraint \\"inv_caboose_pkey\\"") transaction = inventory_insert_collection
75792025-04-30T06:43:00.024ZWARNtest_snapshot_basic (BackgroundTasks): inventory collection failed background_task = inventory_collection error = failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint "inv_caboose_pkey"
75802025-04-30T06:43:00.024ZDEBGtest_snapshot_basic (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 635.390323ms iteration = 23 status = {"error":"failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint \\"inv_caboose_pkey\\""}
75812025-04-30T06:43:00.024ZDEBGtest_snapshot_basic (BackgroundTasks): activating background_task = inventory_collection iteration = 24 reason = Signaled
75822025-04-30T06:43:00.024ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
75832025-04-30T06:43:00.024ZTRACtest_snapshot_basic (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
75842025-04-30T06:43:00.024ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
75852025-04-30T06:43:00.024ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
75862025-04-30T06:43:00.025ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
75872025-04-30T06:43:00.034ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
75882025-04-30T06:43:00.048ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
75892025-04-30T06:43:00.048ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
75902025-04-30T06:43:00.048ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
75912025-04-30T06:43:00.050ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
75922025-04-30T06:43:00.064ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
75932025-04-30T06:43:00.064ZTRACtest_snapshot_basic (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
75942025-04-30T06:43:00.064ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
75952025-04-30T06:43:00.064ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {} }
75962025-04-30T06:43:00.066ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
75972025-04-30T06:43:00.066ZDEBGtest_snapshot_basic (results-sink): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
75982025-04-30T06:43:00.068ZDEBGtest_snapshot_basic (results-sink-cluster): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
75992025-04-30T06:43:00.068ZDEBGtest_snapshot_basic (clickhouse-client): unrolling 1 total samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 6ad06c6b-9a6e-4ed1-b3f4-5d2e9da3f8a7
76002025-04-30T06:43:00.070ZTRACtest_snapshot_basic (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 6ad06c6b-9a6e-4ed1-b3f4-5d2e9da3f8a7 n_columns = 4 n_rows = 2 sql = INSERT INTO oximeter.fields_string FORMAT Native
76012025-04-30T06:43:00.074ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
76022025-04-30T06:43:00.087ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
76032025-04-30T06:43:00.087ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
76042025-04-30T06:43:00.087ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {} }
76052025-04-30T06:43:00.089ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
76062025-04-30T06:43:00.089ZDEBGtest_snapshot_basic (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 6ad06c6b-9a6e-4ed1-b3f4-5d2e9da3f8a7 n_rows = 2 table_name = fields_string
76072025-04-30T06:43:00.089ZTRACtest_snapshot_basic (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 6ad06c6b-9a6e-4ed1-b3f4-5d2e9da3f8a7 n_columns = 4 n_rows = 1 sql = INSERT INTO oximeter.measurements_i64 FORMAT Native
76082025-04-30T06:43:00.090ZDEBGtest_snapshot_basic (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = [(d1ad44e5-18dd-4193-a70e-bf9042f0e0e1, 0)]
76092025-04-30T06:43:00.090ZTRACtest_snapshot_basic (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
76102025-04-30T06:43:00.093ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
76112025-04-30T06:43:00.093ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {} }
76122025-04-30T06:43:00.095ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
76132025-04-30T06:43:00.097ZTRACtest_snapshot_basic (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: f9 dd 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 ............._mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:64821 req_id = c2a7bf13-5928-40ec-954c-5f1821a226e8
76142025-04-30T06:43:00.097ZDEBGtest_snapshot_basic (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 63965,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:64821 req_id = c2a7bf13-5928-40ec-954c-5f1821a226e8
76152025-04-30T06:43:00.097ZDEBGtest_snapshot_basic (store): query key key = _mgs._tcp
76162025-04-30T06:43:00.097ZDEBGtest_snapshot_basic (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
76172025-04-30T06:43:00.097ZDEBGtest_snapshot_basic (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:64821 query = LowerQuery { name: LowerName(Name("_mgs._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 41502, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = c2a7bf13-5928-40ec-954c-5f1821a226e8
76182025-04-30T06:43:00.098ZDEBGtest_snapshot_basic (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 6ad06c6b-9a6e-4ed1-b3f4-5d2e9da3f8a7 n_rows = 1 table_name = measurements_i64
76192025-04-30T06:43:00.098ZTRACtest_snapshot_basic (results-sink): successfully inserted samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
76202025-04-30T06:43:00.099ZDEBGtest_snapshot_basic (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 41502, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 4244, tv_nsec: 898346993 } })
76212025-04-30T06:43:00.099ZTRACtest_snapshot_basic (DnsResolver): lookup_service_targets: looking up SRV target
76222025-04-30T06:43:00.100ZTRACtest_snapshot_basic (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: 5c 28 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e \\(..........-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:58903 req_id = ba0b4b72-0147-4407-b46a-1a2ea9f1abbd
76232025-04-30T06:43:00.100ZDEBGtest_snapshot_basic (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 23592,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:58903 req_id = ba0b4b72-0147-4407-b46a-1a2ea9f1abbd
76242025-04-30T06:43:00.100ZDEBGtest_snapshot_basic (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
76252025-04-30T06:43:00.100ZDEBGtest_snapshot_basic (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:58903 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = ba0b4b72-0147-4407-b46a-1a2ea9f1abbd
76262025-04-30T06:43:00.155ZTRACtest_snapshot_basic (DnsResolver): lookup_all_socket_v6 srv dns_name = _clickhouse-admin-keeper._tcp.control-plane.oxide.internal
76272025-04-30T06:43:00.156ZTRACtest_snapshot_basic (dns): buffer buffer = Length: 87 (0x57) bytes\n0000: ea c0 01 00 00 01 00 00 00 00 00 01 18 5f 63 6c ............._cl\n0010: 69 63 6b 68 6f 75 73 65 2d 61 64 6d 69 6e 2d 6b ickhouse-admin-k\n0020: 65 65 70 65 72 04 5f 74 63 70 0d 63 6f 6e 74 72 eeper._tcp.contr\n0030: 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 69 ol-plane.oxide.i\n0040: 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 04 nternal..!....).\n0050: d0 00 00 00 00 00 00 ....... kind = dns peer_addr = [::1]:53842 req_id = 9ad5572d-23e0-4eb6-81f5-b03981c5b37e
76282025-04-30T06:43:00.156ZDEBGtest_snapshot_basic (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 60096,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 24,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 97,\n 100,\n 109,\n 105,\n 110,\n 45,\n 107,\n 101,\n 101,\n 112,\n 101,\n 114,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:53842 req_id = 9ad5572d-23e0-4eb6-81f5-b03981c5b37e
76292025-04-30T06:43:00.157ZDEBGtest_snapshot_basic (store): query key key = _clickhouse-admin-keeper._tcp
76302025-04-30T06:43:00.157ZERROtest_snapshot_basic (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 60096, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } NXDOMAIN: no records found for name: "_clickhouse-admin-keeper._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:53842 req_id = 9ad5572d-23e0-4eb6-81f5-b03981c5b37e
76312025-04-30T06:43:00.157ZDEBGtest_snapshot_basic (BackgroundTasks): begin collection background_task = inventory_collection
76322025-04-30T06:43:00.157ZDEBGtest_snapshot_basic (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:41502
76332025-04-30T06:43:00.157ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/ignition
76342025-04-30T06:43:00.160ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "38f97356-6447-4ec1-9a54-bb64b5fcd688", "content-length": "835", "date": "Wed, 30 Apr 2025 06:42:59 GMT"} })
76352025-04-30T06:43:00.160ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/0
76362025-04-30T06:43:00.162ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "3a408df1-ba23-408b-a845-bcd1fb539d8f", "content-length": "734", "date": "Wed, 30 Apr 2025 06:42:59 GMT"} })
76372025-04-30T06:43:00.163ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/0/component/sp/caboose?firmware_slot=0
76382025-04-30T06:43:00.168ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "5eb0a36d-7b62-486b-b724-624f966a04c5", "content-length": "111", "date": "Wed, 30 Apr 2025 06:42:59 GMT"} })
76392025-04-30T06:43:00.168ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/0/component/sp/caboose?firmware_slot=1
76402025-04-30T06:43:00.171ZWARNtest_snapshot_basic (results-sink-cluster): failed to insert some results into metric DB collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 error = Failed to check out connection to database: No backends found for this service file = oximeter/collector/src/results_sink.rs:92
76412025-04-30T06:43:00.171ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "9841f258-e4da-4a11-9ff0-00e9eb3a10a2", "content-length": "111", "date": "Wed, 30 Apr 2025 06:42:59 GMT"} })
76422025-04-30T06:43:00.171ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/0/component/rot/caboose?firmware_slot=0
76432025-04-30T06:43:00.177ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "ce190acd-bef4-4a23-86be-3b24f82816fc", "content-length": "170", "date": "Wed, 30 Apr 2025 06:42:59 GMT"} })
76442025-04-30T06:43:00.177ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/0/component/rot/caboose?firmware_slot=1
76452025-04-30T06:43:00.181ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "3cac161f-dbbc-481f-9811-d8e600a1c8a5", "content-length": "170", "date": "Wed, 30 Apr 2025 06:42:59 GMT"} })
76462025-04-30T06:43:00.181ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/0/component/stage0/caboose?firmware_slot=0
76472025-04-30T06:43:00.183ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "a1aba484-44ca-4612-a179-38796515eb63", "content-length": "179", "date": "Wed, 30 Apr 2025 06:42:59 GMT"} })
76482025-04-30T06:43:00.183ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/0/component/stage0/caboose?firmware_slot=1
76492025-04-30T06:43:00.187ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "73c376b7-207e-459b-9f17-152b57c3f94f", "content-length": "179", "date": "Wed, 30 Apr 2025 06:42:59 GMT"} })
76502025-04-30T06:43:00.187ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/0/component/rot/cmpa
76512025-04-30T06:43:00.188ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "440927b6-a114-4922-8027-729b21268f38", "content-length": "702", "date": "Wed, 30 Apr 2025 06:42:59 GMT"} })
76522025-04-30T06:43:00.188ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/0/component/rot/cfpa
76532025-04-30T06:43:00.190ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "c571e201-92c5-4e3d-b9aa-a7630372c7d1", "content-length": "727", "date": "Wed, 30 Apr 2025 06:42:59 GMT"} })
76542025-04-30T06:43:00.190ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/0/component/rot/cfpa
76552025-04-30T06:43:00.193ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "965cc232-934f-4508-8faa-290bb721413a", "content-length": "729", "date": "Wed, 30 Apr 2025 06:42:59 GMT"} })
76562025-04-30T06:43:00.193ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/0/component/rot/cfpa
76572025-04-30T06:43:00.194ZDEBGtest_snapshot_basic: attempt to fetch latest inventory collection result = Ok("found")
76582025-04-30T06:43:00.200ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "21e8a551-31a3-4641-bb51-d9358abe63fa", "content-length": "728", "date": "Wed, 30 Apr 2025 06:42:59 GMT"} })
76592025-04-30T06:43:00.200ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/1
76602025-04-30T06:43:00.212ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "82be18eb-cb5a-47be-bf49-70af86f34962", "content-length": "734", "date": "Wed, 30 Apr 2025 06:42:59 GMT"} })
76612025-04-30T06:43:00.213ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/1/component/sp/caboose?firmware_slot=0
76622025-04-30T06:43:00.219ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "e4d5df31-87e4-4b36-9031-cdf0103f89ff", "content-length": "111", "date": "Wed, 30 Apr 2025 06:42:59 GMT"} })
76632025-04-30T06:43:00.219ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/1/component/sp/caboose?firmware_slot=1
76642025-04-30T06:43:00.223ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "663b94ce-3105-434f-a0ed-4d44ee8265e2", "content-length": "111", "date": "Wed, 30 Apr 2025 06:42:59 GMT"} })
76652025-04-30T06:43:00.224ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/1/component/rot/caboose?firmware_slot=0
76662025-04-30T06:43:00.227ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "a3767571-3149-4697-9c71-09e7f24ed762", "content-length": "170", "date": "Wed, 30 Apr 2025 06:42:59 GMT"} })
76672025-04-30T06:43:00.227ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/1/component/rot/caboose?firmware_slot=1
76682025-04-30T06:43:00.229ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "a8186bac-fe2b-43f4-b3d3-a988b781c57f", "content-length": "170", "date": "Wed, 30 Apr 2025 06:42:59 GMT"} })
76692025-04-30T06:43:00.230ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/1/component/stage0/caboose?firmware_slot=0
76702025-04-30T06:43:00.233ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "6e8b6a72-9af7-4187-9b87-b6e097e1cacd", "content-length": "179", "date": "Wed, 30 Apr 2025 06:42:59 GMT"} })
76712025-04-30T06:43:00.234ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/1/component/stage0/caboose?firmware_slot=1
76722025-04-30T06:43:00.238ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "3423a881-dc12-4306-8154-b8dfe26fbe13", "content-length": "179", "date": "Wed, 30 Apr 2025 06:42:59 GMT"} })
76732025-04-30T06:43:00.238ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/1/component/rot/cmpa
76742025-04-30T06:43:00.240ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "e5885b8b-87a0-43ae-9121-febfcd4bdffc", "content-length": "702", "date": "Wed, 30 Apr 2025 06:42:59 GMT"} })
76752025-04-30T06:43:00.240ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/1/component/rot/cfpa
76762025-04-30T06:43:00.241ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "eb86cc76-5fc1-40b4-b8a3-a1b0357c026a", "content-length": "727", "date": "Wed, 30 Apr 2025 06:42:59 GMT"} })
76772025-04-30T06:43:00.242ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/1/component/rot/cfpa
76782025-04-30T06:43:00.242ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "5708ff95-6f15-43d2-886f-6fd952fa6c59", "content-length": "729", "date": "Wed, 30 Apr 2025 06:42:59 GMT"} })
76792025-04-30T06:43:00.242ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/1/component/rot/cfpa
76802025-04-30T06:43:00.244ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "e7041927-3ac8-49ce-ae4a-487b30665223", "content-length": "728", "date": "Wed, 30 Apr 2025 06:42:59 GMT"} })
76812025-04-30T06:43:00.244ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/0
76822025-04-30T06:43:00.245ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
76832025-04-30T06:43:00.245ZTRACtest_snapshot_basic (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
76842025-04-30T06:43:00.245ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
76852025-04-30T06:43:00.245ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {} }
76862025-04-30T06:43:00.246ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
76872025-04-30T06:43:00.247ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "8f67b602-b604-47ac-a0e9-0bd1ad59cfd7", "content-length": "723", "date": "Wed, 30 Apr 2025 06:42:59 GMT"} })
76882025-04-30T06:43:00.247ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/0/component/sp/caboose?firmware_slot=0
76892025-04-30T06:43:00.249ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
76902025-04-30T06:43:00.257ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
76912025-04-30T06:43:00.257ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
76922025-04-30T06:43:00.257ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {} }
76932025-04-30T06:43:00.258ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
76942025-04-30T06:43:00.261ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "f70fa096-8c72-457c-8cba-96f513373c4d", "content-length": "109", "date": "Wed, 30 Apr 2025 06:42:59 GMT"} })
76952025-04-30T06:43:00.261ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/0/component/sp/caboose?firmware_slot=1
76962025-04-30T06:43:00.261ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
76972025-04-30T06:43:00.261ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {} }
76982025-04-30T06:43:00.262ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
76992025-04-30T06:43:00.264ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "762d3dc1-ce67-4855-9729-0f269606de1c", "content-length": "109", "date": "Wed, 30 Apr 2025 06:42:59 GMT"} })
77002025-04-30T06:43:00.264ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/0/component/rot/caboose?firmware_slot=0
77012025-04-30T06:43:00.267ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "a97fcd8f-2d20-4d3f-bd47-426634446294", "content-length": "169", "date": "Wed, 30 Apr 2025 06:42:59 GMT"} })
77022025-04-30T06:43:00.267ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/0/component/rot/caboose?firmware_slot=1
77032025-04-30T06:43:00.272ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "366d4aaf-810b-446e-898e-9427cd228ef6", "content-length": "169", "date": "Wed, 30 Apr 2025 06:42:59 GMT"} })
77042025-04-30T06:43:00.272ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/0/component/stage0/caboose?firmware_slot=0
77052025-04-30T06:43:00.278ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "ca90b72d-a803-4226-a04e-833011ba09f2", "content-length": "178", "date": "Wed, 30 Apr 2025 06:42:59 GMT"} })
77062025-04-30T06:43:00.279ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/0/component/stage0/caboose?firmware_slot=1
77072025-04-30T06:43:00.281ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "4d2ee2ea-7c2d-489c-9696-8b9e31a193b2", "content-length": "178", "date": "Wed, 30 Apr 2025 06:42:59 GMT"} })
77082025-04-30T06:43:00.281ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/0/component/rot/cmpa
77092025-04-30T06:43:00.282ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "15484594-9b2f-4fc0-ad60-d7587dbdfef9", "content-length": "702", "date": "Wed, 30 Apr 2025 06:42:59 GMT"} })
77102025-04-30T06:43:00.288ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/0/component/rot/cfpa
77112025-04-30T06:43:00.289ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "69e87833-ba81-42b7-869a-1a5c35d1059b", "content-length": "727", "date": "Wed, 30 Apr 2025 06:42:59 GMT"} })
77122025-04-30T06:43:00.291ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/0/component/rot/cfpa
77132025-04-30T06:43:00.293ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "c3bfd140-7865-4ea3-a16b-e848111358dd", "content-length": "729", "date": "Wed, 30 Apr 2025 06:42:59 GMT"} })
77142025-04-30T06:43:00.293ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/0/component/rot/cfpa
77152025-04-30T06:43:00.296ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "3d4bf85f-b241-490f-9b79-6eaf15271708", "content-length": "728", "date": "Wed, 30 Apr 2025 06:42:59 GMT"} })
77162025-04-30T06:43:00.297ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/1
77172025-04-30T06:43:00.298ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "32c673ad-0676-4572-b6dd-4bfc43af8616", "content-length": "723", "date": "Wed, 30 Apr 2025 06:42:59 GMT"} })
77182025-04-30T06:43:00.298ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/1/component/sp/caboose?firmware_slot=0
77192025-04-30T06:43:00.303ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "d5c9d83c-9e77-421b-9228-164523baa8ba", "content-length": "109", "date": "Wed, 30 Apr 2025 06:42:59 GMT"} })
77202025-04-30T06:43:00.303ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/1/component/sp/caboose?firmware_slot=1
77212025-04-30T06:43:00.306ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "b95bb798-30ca-4bf0-be65-37e217aec2a2", "content-length": "109", "date": "Wed, 30 Apr 2025 06:42:59 GMT"} })
77222025-04-30T06:43:00.307ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/1/component/rot/caboose?firmware_slot=0
77232025-04-30T06:43:00.310ZDEBGtest_snapshot_basic: attempt to fetch latest inventory collection result = Ok("found")
77242025-04-30T06:43:00.312ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "181ac440-0365-47ff-ac9f-88c95bd14bd3", "content-length": "169", "date": "Wed, 30 Apr 2025 06:42:59 GMT"} })
77252025-04-30T06:43:00.312ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/1/component/rot/caboose?firmware_slot=1
77262025-04-30T06:43:00.314ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "9d0858c3-137f-445a-82c6-5eaafc722f3e", "content-length": "169", "date": "Wed, 30 Apr 2025 06:42:59 GMT"} })
77272025-04-30T06:43:00.314ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/1/component/stage0/caboose?firmware_slot=0
77282025-04-30T06:43:00.316ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "ccdbeef0-863a-4647-a2a6-24e96fbaf159", "content-length": "178", "date": "Wed, 30 Apr 2025 06:42:59 GMT"} })
77292025-04-30T06:43:00.316ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/1/component/stage0/caboose?firmware_slot=1
77302025-04-30T06:43:00.318ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "715b5898-b33f-47c0-afe3-eeed560943ef", "content-length": "178", "date": "Wed, 30 Apr 2025 06:42:59 GMT"} })
77312025-04-30T06:43:00.318ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/1/component/rot/cmpa
77322025-04-30T06:43:00.319ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "546410f4-ebd4-4fb2-9288-852a08df35f5", "content-length": "702", "date": "Wed, 30 Apr 2025 06:42:59 GMT"} })
77332025-04-30T06:43:00.320ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/1/component/rot/cfpa
77342025-04-30T06:43:00.321ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "7c23d619-f793-46b2-95a8-e7d7cf87965d", "content-length": "727", "date": "Wed, 30 Apr 2025 06:42:59 GMT"} })
77352025-04-30T06:43:00.321ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/1/component/rot/cfpa
77362025-04-30T06:43:00.322ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "80614cf2-8651-4267-b186-41463998d7ea", "content-length": "729", "date": "Wed, 30 Apr 2025 06:42:59 GMT"} })
77372025-04-30T06:43:00.322ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/1/component/rot/cfpa
77382025-04-30T06:43:00.323ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "63a4f70e-5335-40ee-9a1e-61baf6dc9194", "content-length": "728", "date": "Wed, 30 Apr 2025 06:42:59 GMT"} })
77392025-04-30T06:43:00.323ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
77402025-04-30T06:43:00.323ZTRACtest_snapshot_basic (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
77412025-04-30T06:43:00.323ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
77422025-04-30T06:43:00.324ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
77432025-04-30T06:43:00.324ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
77442025-04-30T06:43:00.326ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
77452025-04-30T06:43:00.332ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
77462025-04-30T06:43:00.333ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
77472025-04-30T06:43:00.333ZTRACtest_snapshot_basic (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
77482025-04-30T06:43:00.333ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
77492025-04-30T06:43:00.333ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
77502025-04-30T06:43:00.333ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
77512025-04-30T06:43:00.336ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
77522025-04-30T06:43:00.340ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
77532025-04-30T06:43:00.340ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
77542025-04-30T06:43:00.340ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
77552025-04-30T06:43:00.341ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
77562025-04-30T06:43:00.393ZDEBGtest_snapshot_basic (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:41923
77572025-04-30T06:43:00.393ZDEBGtest_snapshot_basic (BackgroundTasks): client request SledAgent = http://[::1]:41923 background_task = inventory_collection body = None method = GET uri = http://[::1]:41923/inventory
77582025-04-30T06:43:00.393ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
77592025-04-30T06:43:00.393ZTRACtest_snapshot_basic (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
77602025-04-30T06:43:00.393ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
77612025-04-30T06:43:00.393ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {} }
77622025-04-30T06:43:00.394ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
77632025-04-30T06:43:00.394ZINFOtest_snapshot_basic (dropshot): accepted connection local_addr = [::1]:41923 remote_addr = [::1]:42174 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
77642025-04-30T06:43:00.396ZTRACtest_snapshot_basic (dropshot): incoming request local_addr = [::1]:41923 method = GET remote_addr = [::1]:42174 req_id = 63fb6116-19c6-44d1-ada7-45e5350df92f sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
77652025-04-30T06:43:00.397ZINFOtest_snapshot_basic (dropshot): request completed latency_us = 621 local_addr = [::1]:41923 method = GET remote_addr = [::1]:42174 req_id = 63fb6116-19c6-44d1-ada7-45e5350df92f response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
77662025-04-30T06:43:00.398ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
77672025-04-30T06:43:00.407ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
77682025-04-30T06:43:00.407ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
77692025-04-30T06:43:00.407ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {} }
77702025-04-30T06:43:00.408ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
77712025-04-30T06:43:00.409ZDEBGtest_snapshot_basic (BackgroundTasks): client response SledAgent = http://[::1]:41923 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:41923/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "63fb6116-19c6-44d1-ada7-45e5350df92f", "content-length": "3141", "date": "Wed, 30 Apr 2025 06:42:59 GMT"} })
77722025-04-30T06:43:00.413ZDEBGtest_snapshot_basic (BackgroundTasks): begin collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
77732025-04-30T06:43:00.414ZDEBGtest_snapshot_basic (BackgroundTasks): end collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
77742025-04-30T06:43:00.414ZDEBGtest_snapshot_basic (BackgroundTasks): finished collection background_task = inventory_collection
77752025-04-30T06:43:00.415ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
77762025-04-30T06:43:00.415ZTRACtest_snapshot_basic (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
77772025-04-30T06:43:00.416ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
77782025-04-30T06:43:00.416ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
77792025-04-30T06:43:00.416ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
77802025-04-30T06:43:00.418ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
77812025-04-30T06:43:00.418ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {} }
77822025-04-30T06:43:00.419ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
77832025-04-30T06:43:00.420ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
77842025-04-30T06:43:00.440ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
77852025-04-30T06:43:00.440ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
77862025-04-30T06:43:00.440ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
77872025-04-30T06:43:00.441ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
77882025-04-30T06:43:00.495ZDEBGtest_snapshot_basic: attempt to fetch latest inventory collection result = Ok("found")
77892025-04-30T06:43:00.498ZWARNtest_snapshot_basic (datastore): Non-retryable transaction failure err = DatabaseError(UniqueViolation, "duplicate key value violates unique constraint \\"inv_caboose_pkey\\"") transaction = inventory_insert_collection
77902025-04-30T06:43:00.498ZWARNtest_snapshot_basic (BackgroundTasks): inventory collection failed background_task = inventory_collection error = failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint "inv_caboose_pkey"
77912025-04-30T06:43:00.498ZDEBGtest_snapshot_basic (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 474.162147ms iteration = 24 status = {"error":"failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint \\"inv_caboose_pkey\\""}
77922025-04-30T06:43:00.498ZDEBGtest_snapshot_basic (BackgroundTasks): activating background_task = inventory_collection iteration = 25 reason = Signaled
77932025-04-30T06:43:00.498ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
77942025-04-30T06:43:00.500ZTRACtest_snapshot_basic (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
77952025-04-30T06:43:00.500ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
77962025-04-30T06:43:00.500ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
77972025-04-30T06:43:00.502ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
77982025-04-30T06:43:00.504ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
77992025-04-30T06:43:00.509ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
78002025-04-30T06:43:00.509ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
78012025-04-30T06:43:00.509ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
78022025-04-30T06:43:00.510ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
78032025-04-30T06:43:00.514ZDEBGtest_snapshot_basic (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = [(d1ad44e5-18dd-4193-a70e-bf9042f0e0e1, 0)]
78042025-04-30T06:43:00.514ZTRACtest_snapshot_basic (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
78052025-04-30T06:43:00.523ZTRACtest_snapshot_basic (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: 82 c1 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 ............._mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:51602 req_id = 3970e793-3f13-4279-b3fa-088d90925003
78062025-04-30T06:43:00.523ZDEBGtest_snapshot_basic (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 33473,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:51602 req_id = 3970e793-3f13-4279-b3fa-088d90925003
78072025-04-30T06:43:00.524ZDEBGtest_snapshot_basic (store): query key key = _mgs._tcp
78082025-04-30T06:43:00.524ZDEBGtest_snapshot_basic (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
78092025-04-30T06:43:00.524ZDEBGtest_snapshot_basic (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:51602 query = LowerQuery { name: LowerName(Name("_mgs._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 41502, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = 3970e793-3f13-4279-b3fa-088d90925003
78102025-04-30T06:43:00.527ZDEBGtest_snapshot_basic (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 41502, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 4245, tv_nsec: 325905256 } })
78112025-04-30T06:43:00.527ZTRACtest_snapshot_basic (DnsResolver): lookup_service_targets: looking up SRV target
78122025-04-30T06:43:00.532ZTRACtest_snapshot_basic (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: c5 a8 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e ............-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:51225 req_id = 7c400d0b-3dd8-4b93-9dbc-ffd226282dae
78132025-04-30T06:43:00.532ZDEBGtest_snapshot_basic (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 50600,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:51225 req_id = 7c400d0b-3dd8-4b93-9dbc-ffd226282dae
78142025-04-30T06:43:00.532ZDEBGtest_snapshot_basic (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
78152025-04-30T06:43:00.534ZDEBGtest_snapshot_basic (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:51225 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = 7c400d0b-3dd8-4b93-9dbc-ffd226282dae
78162025-04-30T06:43:00.633ZTRACtest_snapshot_basic (DnsResolver): lookup_all_socket_v6 srv dns_name = _clickhouse-admin-keeper._tcp.control-plane.oxide.internal
78172025-04-30T06:43:00.635ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
78182025-04-30T06:43:00.635ZTRACtest_snapshot_basic (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
78192025-04-30T06:43:00.635ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
78202025-04-30T06:43:00.636ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {} }
78212025-04-30T06:43:00.636ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
78222025-04-30T06:43:00.639ZTRACtest_snapshot_basic (dns): buffer buffer = Length: 87 (0x57) bytes\n0000: 4e 03 01 00 00 01 00 00 00 00 00 01 18 5f 63 6c N............_cl\n0010: 69 63 6b 68 6f 75 73 65 2d 61 64 6d 69 6e 2d 6b ickhouse-admin-k\n0020: 65 65 70 65 72 04 5f 74 63 70 0d 63 6f 6e 74 72 eeper._tcp.contr\n0030: 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 69 ol-plane.oxide.i\n0040: 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 04 nternal..!....).\n0050: d0 00 00 00 00 00 00 ....... kind = dns peer_addr = [::1]:51578 req_id = b365dcb2-fcba-4dfc-8e5a-467b16f499c3
78232025-04-30T06:43:00.639ZDEBGtest_snapshot_basic (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 19971,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 24,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 97,\n 100,\n 109,\n 105,\n 110,\n 45,\n 107,\n 101,\n 101,\n 112,\n 101,\n 114,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:51578 req_id = b365dcb2-fcba-4dfc-8e5a-467b16f499c3
78242025-04-30T06:43:00.639ZDEBGtest_snapshot_basic (store): query key key = _clickhouse-admin-keeper._tcp
78252025-04-30T06:43:00.639ZERROtest_snapshot_basic (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 19971, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } NXDOMAIN: no records found for name: "_clickhouse-admin-keeper._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:51578 req_id = b365dcb2-fcba-4dfc-8e5a-467b16f499c3
78262025-04-30T06:43:00.640ZDEBGtest_snapshot_basic (BackgroundTasks): begin collection background_task = inventory_collection
78272025-04-30T06:43:00.640ZDEBGtest_snapshot_basic (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:41502
78282025-04-30T06:43:00.640ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/ignition
78292025-04-30T06:43:00.646ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
78302025-04-30T06:43:00.656ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
78312025-04-30T06:43:00.656ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
78322025-04-30T06:43:00.656ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {} }
78332025-04-30T06:43:00.657ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
78342025-04-30T06:43:00.666ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "859b66fc-62a9-4468-8718-f816a5ed6b22", "content-length": "835", "date": "Wed, 30 Apr 2025 06:43:00 GMT"} })
78352025-04-30T06:43:00.667ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/0
78362025-04-30T06:43:00.670ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "2313e771-a362-4c93-8e16-00bbda96a6ca", "content-length": "734", "date": "Wed, 30 Apr 2025 06:43:00 GMT"} })
78372025-04-30T06:43:00.670ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/0/component/sp/caboose?firmware_slot=0
78382025-04-30T06:43:00.670ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
78392025-04-30T06:43:00.670ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {} }
78402025-04-30T06:43:00.672ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
78412025-04-30T06:43:00.687ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "a72410d0-e5f4-4dc0-90b2-80bfa23a1277", "content-length": "111", "date": "Wed, 30 Apr 2025 06:43:00 GMT"} })
78422025-04-30T06:43:00.688ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/0/component/sp/caboose?firmware_slot=1
78432025-04-30T06:43:00.707ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "a23993df-9d9f-4f97-a708-425a54ec6efc", "content-length": "111", "date": "Wed, 30 Apr 2025 06:43:00 GMT"} })
78442025-04-30T06:43:00.707ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/0/component/rot/caboose?firmware_slot=0
78452025-04-30T06:43:00.713ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "350fb1a8-6686-48a7-834f-94f66beef23d", "content-length": "170", "date": "Wed, 30 Apr 2025 06:43:00 GMT"} })
78462025-04-30T06:43:00.713ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/0/component/rot/caboose?firmware_slot=1
78472025-04-30T06:43:00.721ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "cc643c64-2876-40fc-a2cb-52b5d5156088", "content-length": "170", "date": "Wed, 30 Apr 2025 06:43:00 GMT"} })
78482025-04-30T06:43:00.721ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/0/component/stage0/caboose?firmware_slot=0
78492025-04-30T06:43:00.728ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "0ea56177-a6ba-4b46-b4f3-88d4dcf0f64a", "content-length": "179", "date": "Wed, 30 Apr 2025 06:43:00 GMT"} })
78502025-04-30T06:43:00.728ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/0/component/stage0/caboose?firmware_slot=1
78512025-04-30T06:43:00.732ZDEBGtest_snapshot_basic: attempt to fetch latest inventory collection result = Ok("found")
78522025-04-30T06:43:00.735ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "f9ebd545-220a-4fdc-bc8e-2d89130414d4", "content-length": "179", "date": "Wed, 30 Apr 2025 06:43:00 GMT"} })
78532025-04-30T06:43:00.735ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/0/component/rot/cmpa
78542025-04-30T06:43:00.736ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "93837000-4b97-44c7-8416-38c965c26a42", "content-length": "702", "date": "Wed, 30 Apr 2025 06:43:00 GMT"} })
78552025-04-30T06:43:00.736ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/0/component/rot/cfpa
78562025-04-30T06:43:00.738ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "e86d56c1-efe9-457f-b81a-1c65ddacbfc3", "content-length": "727", "date": "Wed, 30 Apr 2025 06:43:00 GMT"} })
78572025-04-30T06:43:00.738ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/0/component/rot/cfpa
78582025-04-30T06:43:00.739ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "0f0dfc6a-5bd4-4682-b3b6-215cb75bd30b", "content-length": "729", "date": "Wed, 30 Apr 2025 06:43:00 GMT"} })
78592025-04-30T06:43:00.739ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/0/component/rot/cfpa
78602025-04-30T06:43:00.740ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "3ac3a869-6230-413c-85e0-27cca6899bd6", "content-length": "728", "date": "Wed, 30 Apr 2025 06:43:00 GMT"} })
78612025-04-30T06:43:00.740ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/1
78622025-04-30T06:43:00.743ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "2e9c04e6-a829-4dce-b4f8-febbf7f88780", "content-length": "734", "date": "Wed, 30 Apr 2025 06:43:00 GMT"} })
78632025-04-30T06:43:00.743ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/1/component/sp/caboose?firmware_slot=0
78642025-04-30T06:43:00.746ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "04a060f0-714a-4c85-87a3-8a8d4b505677", "content-length": "111", "date": "Wed, 30 Apr 2025 06:43:00 GMT"} })
78652025-04-30T06:43:00.746ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/1/component/sp/caboose?firmware_slot=1
78662025-04-30T06:43:00.749ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "6788a2e6-e613-4117-b506-b1ed316bdffc", "content-length": "111", "date": "Wed, 30 Apr 2025 06:43:00 GMT"} })
78672025-04-30T06:43:00.749ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/1/component/rot/caboose?firmware_slot=0
78682025-04-30T06:43:00.753ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "75cf6fba-5db1-4f65-ab97-4ae87e904829", "content-length": "170", "date": "Wed, 30 Apr 2025 06:43:00 GMT"} })
78692025-04-30T06:43:00.753ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/1/component/rot/caboose?firmware_slot=1
78702025-04-30T06:43:00.756ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "505c54e1-9e9c-4b9d-ba4b-ca491fad18cf", "content-length": "170", "date": "Wed, 30 Apr 2025 06:43:00 GMT"} })
78712025-04-30T06:43:00.757ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/1/component/stage0/caboose?firmware_slot=0
78722025-04-30T06:43:00.763ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "59b49585-bb0a-4177-b560-e117457ad048", "content-length": "179", "date": "Wed, 30 Apr 2025 06:43:00 GMT"} })
78732025-04-30T06:43:00.763ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/1/component/stage0/caboose?firmware_slot=1
78742025-04-30T06:43:00.767ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "96e69e14-3844-4fa8-8785-1d11e28b9b07", "content-length": "179", "date": "Wed, 30 Apr 2025 06:43:00 GMT"} })
78752025-04-30T06:43:00.767ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/1/component/rot/cmpa
78762025-04-30T06:43:00.769ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "9af2bfca-5227-4ccc-b56d-bfa450fe44d7", "content-length": "702", "date": "Wed, 30 Apr 2025 06:43:00 GMT"} })
78772025-04-30T06:43:00.770ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/1/component/rot/cfpa
78782025-04-30T06:43:00.772ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "b846761b-e0f4-4a50-b398-4b92e4821e3c", "content-length": "727", "date": "Wed, 30 Apr 2025 06:43:00 GMT"} })
78792025-04-30T06:43:00.772ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/1/component/rot/cfpa
78802025-04-30T06:43:00.773ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "4f8c1603-196f-4a9f-9885-dcaf2e84f60b", "content-length": "729", "date": "Wed, 30 Apr 2025 06:43:00 GMT"} })
78812025-04-30T06:43:00.773ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/1/component/rot/cfpa
78822025-04-30T06:43:00.775ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "dd13c8d8-16d1-46f1-88f0-a1f4cb05c8e6", "content-length": "728", "date": "Wed, 30 Apr 2025 06:43:00 GMT"} })
78832025-04-30T06:43:00.775ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/0
78842025-04-30T06:43:00.776ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "bc4f7e9e-bb68-4c88-a08f-0ef965941c07", "content-length": "723", "date": "Wed, 30 Apr 2025 06:43:00 GMT"} })
78852025-04-30T06:43:00.776ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/0/component/sp/caboose?firmware_slot=0
78862025-04-30T06:43:00.793ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
78872025-04-30T06:43:00.793ZTRACtest_snapshot_basic (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
78882025-04-30T06:43:00.793ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
78892025-04-30T06:43:00.793ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {} }
78902025-04-30T06:43:00.794ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
78912025-04-30T06:43:00.795ZDEBGtest_snapshot_basic (collection-task): sent timer-based collection request to the collection task collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
78922025-04-30T06:43:00.795ZDEBGtest_snapshot_basic (collection-task): collecting from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
78932025-04-30T06:43:00.797ZTRACtest_snapshot_basic (collection-task): sent collection request to producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
78942025-04-30T06:43:00.798ZDEBGtest_snapshot_basic (collection-task): collected results from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 n_results = 1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
78952025-04-30T06:43:00.798ZTRACtest_snapshot_basic (collection-task): forwarded results to main collection loop collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
78962025-04-30T06:43:00.798ZTRACtest_snapshot_basic (collection-task): top of inner collection loop, waiting for next request collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
78972025-04-30T06:43:00.798ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
78982025-04-30T06:43:00.805ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
78992025-04-30T06:43:00.805ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
79002025-04-30T06:43:00.805ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {} }
79012025-04-30T06:43:00.806ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
79022025-04-30T06:43:00.809ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "29e2a084-770b-4b99-abd8-e1d98d8053d9", "content-length": "109", "date": "Wed, 30 Apr 2025 06:43:00 GMT"} })
79032025-04-30T06:43:00.809ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/0/component/sp/caboose?firmware_slot=1
79042025-04-30T06:43:00.810ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
79052025-04-30T06:43:00.810ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {} }
79062025-04-30T06:43:00.810ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
79072025-04-30T06:43:00.813ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "78003fb6-448b-4623-94f9-da9411dd63c2", "content-length": "109", "date": "Wed, 30 Apr 2025 06:43:00 GMT"} })
79082025-04-30T06:43:00.813ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/0/component/rot/caboose?firmware_slot=0
79092025-04-30T06:43:00.815ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "1148b9fa-68d6-44ec-899f-ced05e28ffd5", "content-length": "169", "date": "Wed, 30 Apr 2025 06:43:00 GMT"} })
79102025-04-30T06:43:00.816ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/0/component/rot/caboose?firmware_slot=1
79112025-04-30T06:43:00.818ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "11363b4f-b657-4867-aeda-7d0cb36a10aa", "content-length": "169", "date": "Wed, 30 Apr 2025 06:43:00 GMT"} })
79122025-04-30T06:43:00.818ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/0/component/stage0/caboose?firmware_slot=0
79132025-04-30T06:43:00.820ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "393bb8da-87fb-409f-9bd7-338a22b9f65d", "content-length": "178", "date": "Wed, 30 Apr 2025 06:43:00 GMT"} })
79142025-04-30T06:43:00.820ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/0/component/stage0/caboose?firmware_slot=1
79152025-04-30T06:43:00.822ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "407e87bd-9c2f-4bbc-a112-68d9f61d1732", "content-length": "178", "date": "Wed, 30 Apr 2025 06:43:00 GMT"} })
79162025-04-30T06:43:00.823ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/0/component/rot/cmpa
79172025-04-30T06:43:00.824ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "7017a069-b07a-45f1-a4f3-0e6af635db27", "content-length": "702", "date": "Wed, 30 Apr 2025 06:43:00 GMT"} })
79182025-04-30T06:43:00.824ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/0/component/rot/cfpa
79192025-04-30T06:43:00.834ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "7bf806ce-1de2-42ec-83de-620ad905d927", "content-length": "727", "date": "Wed, 30 Apr 2025 06:43:00 GMT"} })
79202025-04-30T06:43:00.835ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/0/component/rot/cfpa
79212025-04-30T06:43:00.838ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "e8054204-e1c7-4489-b8e3-e9bf17ab7805", "content-length": "729", "date": "Wed, 30 Apr 2025 06:43:00 GMT"} })
79222025-04-30T06:43:00.838ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/0/component/rot/cfpa
79232025-04-30T06:43:00.839ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "a24f168b-ac8e-4cb3-8fd0-ecd583e59bb1", "content-length": "728", "date": "Wed, 30 Apr 2025 06:43:00 GMT"} })
79242025-04-30T06:43:00.839ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/1
79252025-04-30T06:43:00.841ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "d8a2afb9-33cf-45d1-b9b3-5ef460b4ee4b", "content-length": "723", "date": "Wed, 30 Apr 2025 06:43:00 GMT"} })
79262025-04-30T06:43:00.841ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/1/component/sp/caboose?firmware_slot=0
79272025-04-30T06:43:00.846ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "57774ca6-3e5a-4b83-9973-cc61918469a0", "content-length": "109", "date": "Wed, 30 Apr 2025 06:43:00 GMT"} })
79282025-04-30T06:43:00.846ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/1/component/sp/caboose?firmware_slot=1
79292025-04-30T06:43:00.856ZDEBGtest_snapshot_basic: attempt to fetch latest inventory collection result = Ok("found")
79302025-04-30T06:43:00.860ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "57c355d1-fe8f-4ca6-a4b5-fd41f7064877", "content-length": "109", "date": "Wed, 30 Apr 2025 06:43:00 GMT"} })
79312025-04-30T06:43:00.861ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/1/component/rot/caboose?firmware_slot=0
79322025-04-30T06:43:00.864ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "6f0665ac-53d3-4079-9693-9b13cde5c207", "content-length": "169", "date": "Wed, 30 Apr 2025 06:43:00 GMT"} })
79332025-04-30T06:43:00.864ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/1/component/rot/caboose?firmware_slot=1
79342025-04-30T06:43:00.867ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "7bebbfbf-f8be-4bce-ab62-549697ec7509", "content-length": "169", "date": "Wed, 30 Apr 2025 06:43:00 GMT"} })
79352025-04-30T06:43:00.867ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/1/component/stage0/caboose?firmware_slot=0
79362025-04-30T06:43:00.869ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "43c999d8-6896-43a0-bca2-fe0ec8bbfa1c", "content-length": "178", "date": "Wed, 30 Apr 2025 06:43:00 GMT"} })
79372025-04-30T06:43:00.869ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/1/component/stage0/caboose?firmware_slot=1
79382025-04-30T06:43:00.871ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "b880b86b-7778-464d-9687-9f51521c9096", "content-length": "178", "date": "Wed, 30 Apr 2025 06:43:00 GMT"} })
79392025-04-30T06:43:00.871ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/1/component/rot/cmpa
79402025-04-30T06:43:00.872ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "f926ebf9-55e8-45bf-9ec2-7af78f4a008a", "content-length": "702", "date": "Wed, 30 Apr 2025 06:43:00 GMT"} })
79412025-04-30T06:43:00.872ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/1/component/rot/cfpa
79422025-04-30T06:43:00.874ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "d5c6d599-0d89-4dff-8c76-2404618fb4c4", "content-length": "727", "date": "Wed, 30 Apr 2025 06:43:00 GMT"} })
79432025-04-30T06:43:00.875ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/1/component/rot/cfpa
79442025-04-30T06:43:00.877ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "ecea8f90-8103-475e-8fe1-c326642deea3", "content-length": "729", "date": "Wed, 30 Apr 2025 06:43:00 GMT"} })
79452025-04-30T06:43:00.877ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/1/component/rot/cfpa
79462025-04-30T06:43:00.880ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "b347fd71-f721-4715-8c5a-511812bd3286", "content-length": "728", "date": "Wed, 30 Apr 2025 06:43:00 GMT"} })
79472025-04-30T06:43:00.880ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
79482025-04-30T06:43:00.881ZTRACtest_snapshot_basic (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
79492025-04-30T06:43:00.881ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
79502025-04-30T06:43:00.881ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
79512025-04-30T06:43:00.882ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
79522025-04-30T06:43:00.884ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
79532025-04-30T06:43:00.890ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
79542025-04-30T06:43:00.890ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
79552025-04-30T06:43:00.890ZTRACtest_snapshot_basic (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
79562025-04-30T06:43:00.890ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
79572025-04-30T06:43:00.890ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
79582025-04-30T06:43:00.890ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
79592025-04-30T06:43:00.893ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
79602025-04-30T06:43:00.898ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
79612025-04-30T06:43:00.898ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
79622025-04-30T06:43:00.899ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
79632025-04-30T06:43:00.900ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
79642025-04-30T06:43:00.963ZDEBGtest_snapshot_basic (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:41923
79652025-04-30T06:43:00.963ZDEBGtest_snapshot_basic (BackgroundTasks): client request SledAgent = http://[::1]:41923 background_task = inventory_collection body = None method = GET uri = http://[::1]:41923/inventory
79662025-04-30T06:43:00.964ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
79672025-04-30T06:43:00.964ZTRACtest_snapshot_basic (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
79682025-04-30T06:43:00.964ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
79692025-04-30T06:43:00.964ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {} }
79702025-04-30T06:43:00.965ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
79712025-04-30T06:43:00.965ZINFOtest_snapshot_basic (dropshot): accepted connection local_addr = [::1]:41923 remote_addr = [::1]:62456 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
79722025-04-30T06:43:00.966ZTRACtest_snapshot_basic (dropshot): incoming request local_addr = [::1]:41923 method = GET remote_addr = [::1]:62456 req_id = d3704969-28e4-44f2-951a-6bf9e396ddee sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
79732025-04-30T06:43:00.966ZINFOtest_snapshot_basic (dropshot): request completed latency_us = 523 local_addr = [::1]:41923 method = GET remote_addr = [::1]:62456 req_id = d3704969-28e4-44f2-951a-6bf9e396ddee response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
79742025-04-30T06:43:00.968ZDEBGtest_snapshot_basic (BackgroundTasks): client response SledAgent = http://[::1]:41923 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:41923/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "d3704969-28e4-44f2-951a-6bf9e396ddee", "content-length": "3141", "date": "Wed, 30 Apr 2025 06:43:00 GMT"} })
79752025-04-30T06:43:00.970ZDEBGtest_snapshot_basic (BackgroundTasks): begin collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
79762025-04-30T06:43:00.970ZDEBGtest_snapshot_basic (BackgroundTasks): end collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
79772025-04-30T06:43:00.970ZDEBGtest_snapshot_basic (BackgroundTasks): finished collection background_task = inventory_collection
79782025-04-30T06:43:00.974ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
79792025-04-30T06:43:00.974ZTRACtest_snapshot_basic (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
79802025-04-30T06:43:00.974ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
79812025-04-30T06:43:00.974ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
79822025-04-30T06:43:00.975ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
79832025-04-30T06:43:00.976ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
79842025-04-30T06:43:00.983ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
79852025-04-30T06:43:00.983ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
79862025-04-30T06:43:00.983ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {} }
79872025-04-30T06:43:00.983ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
79882025-04-30T06:43:00.985ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
79892025-04-30T06:43:00.989ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
79902025-04-30T06:43:00.990ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
79912025-04-30T06:43:00.990ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
79922025-04-30T06:43:00.991ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
79932025-04-30T06:43:00.991ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
79942025-04-30T06:43:00.991ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {} }
79952025-04-30T06:43:00.992ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
79962025-04-30T06:43:01.021ZDEBGtest_snapshot_basic: attempt to fetch latest inventory collection result = Ok("found")
79972025-04-30T06:43:01.022ZWARNtest_snapshot_basic (datastore): Non-retryable transaction failure err = DatabaseError(UniqueViolation, "duplicate key value violates unique constraint \\"inv_caboose_pkey\\"") transaction = inventory_insert_collection
79982025-04-30T06:43:01.022ZWARNtest_snapshot_basic (BackgroundTasks): inventory collection failed background_task = inventory_collection error = failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint "inv_caboose_pkey"
79992025-04-30T06:43:01.022ZDEBGtest_snapshot_basic (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 523.856951ms iteration = 25 status = {"error":"failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint \\"inv_caboose_pkey\\""}
80002025-04-30T06:43:01.022ZDEBGtest_snapshot_basic (BackgroundTasks): activating background_task = inventory_collection iteration = 26 reason = Signaled
80012025-04-30T06:43:01.022ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
80022025-04-30T06:43:01.022ZTRACtest_snapshot_basic (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
80032025-04-30T06:43:01.022ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
80042025-04-30T06:43:01.022ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
80052025-04-30T06:43:01.023ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
80062025-04-30T06:43:01.026ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
80072025-04-30T06:43:01.030ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
80082025-04-30T06:43:01.030ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
80092025-04-30T06:43:01.030ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
80102025-04-30T06:43:01.031ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
80112025-04-30T06:43:01.035ZDEBGtest_snapshot_basic (results-sink): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
80122025-04-30T06:43:01.036ZDEBGtest_snapshot_basic (results-sink-cluster): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
80132025-04-30T06:43:01.036ZDEBGtest_snapshot_basic (clickhouse-client): unrolling 1 total samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 6ad06c6b-9a6e-4ed1-b3f4-5d2e9da3f8a7
80142025-04-30T06:43:01.037ZTRACtest_snapshot_basic (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 6ad06c6b-9a6e-4ed1-b3f4-5d2e9da3f8a7 n_columns = 4 n_rows = 2 sql = INSERT INTO oximeter.fields_string FORMAT Native
80152025-04-30T06:43:01.037ZDEBGtest_snapshot_basic (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = [(d1ad44e5-18dd-4193-a70e-bf9042f0e0e1, 0)]
80162025-04-30T06:43:01.037ZTRACtest_snapshot_basic (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
80172025-04-30T06:43:01.039ZTRACtest_snapshot_basic (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: 17 1f 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 ............._mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:59657 req_id = 4ceb83d6-e0e9-4a7d-b631-8f734f794f99
80182025-04-30T06:43:01.039ZDEBGtest_snapshot_basic (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 5919,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:59657 req_id = 4ceb83d6-e0e9-4a7d-b631-8f734f794f99
80192025-04-30T06:43:01.040ZDEBGtest_snapshot_basic (store): query key key = _mgs._tcp
80202025-04-30T06:43:01.041ZDEBGtest_snapshot_basic (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
80212025-04-30T06:43:01.041ZDEBGtest_snapshot_basic (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:59657 query = LowerQuery { name: LowerName(Name("_mgs._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 41502, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = 4ceb83d6-e0e9-4a7d-b631-8f734f794f99
80222025-04-30T06:43:01.042ZDEBGtest_snapshot_basic (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 6ad06c6b-9a6e-4ed1-b3f4-5d2e9da3f8a7 n_rows = 2 table_name = fields_string
80232025-04-30T06:43:01.042ZTRACtest_snapshot_basic (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 6ad06c6b-9a6e-4ed1-b3f4-5d2e9da3f8a7 n_columns = 4 n_rows = 1 sql = INSERT INTO oximeter.measurements_i64 FORMAT Native
80242025-04-30T06:43:01.042ZDEBGtest_snapshot_basic (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 41502, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 4245, tv_nsec: 841316779 } })
80252025-04-30T06:43:01.042ZTRACtest_snapshot_basic (DnsResolver): lookup_service_targets: looking up SRV target
80262025-04-30T06:43:01.044ZTRACtest_snapshot_basic (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: c0 4b 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e .K..........-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:65291 req_id = b3064bfd-1551-4f52-a7ae-3d1887e766fc
80272025-04-30T06:43:01.045ZDEBGtest_snapshot_basic (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 49227,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:65291 req_id = b3064bfd-1551-4f52-a7ae-3d1887e766fc
80282025-04-30T06:43:01.045ZDEBGtest_snapshot_basic (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
80292025-04-30T06:43:01.045ZDEBGtest_snapshot_basic (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:65291 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = b3064bfd-1551-4f52-a7ae-3d1887e766fc
80302025-04-30T06:43:01.046ZDEBGtest_snapshot_basic (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 6ad06c6b-9a6e-4ed1-b3f4-5d2e9da3f8a7 n_rows = 1 table_name = measurements_i64
80312025-04-30T06:43:01.046ZTRACtest_snapshot_basic (results-sink): successfully inserted samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
80322025-04-30T06:43:01.100ZTRACtest_snapshot_basic (DnsResolver): lookup_all_socket_v6 srv dns_name = _clickhouse-admin-keeper._tcp.control-plane.oxide.internal
80332025-04-30T06:43:01.101ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
80342025-04-30T06:43:01.101ZTRACtest_snapshot_basic (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
80352025-04-30T06:43:01.101ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
80362025-04-30T06:43:01.101ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {} }
80372025-04-30T06:43:01.102ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
80382025-04-30T06:43:01.102ZTRACtest_snapshot_basic (dns): buffer buffer = Length: 87 (0x57) bytes\n0000: c7 92 01 00 00 01 00 00 00 00 00 01 18 5f 63 6c ............._cl\n0010: 69 63 6b 68 6f 75 73 65 2d 61 64 6d 69 6e 2d 6b ickhouse-admin-k\n0020: 65 65 70 65 72 04 5f 74 63 70 0d 63 6f 6e 74 72 eeper._tcp.contr\n0030: 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 69 ol-plane.oxide.i\n0040: 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 04 nternal..!....).\n0050: d0 00 00 00 00 00 00 ....... kind = dns peer_addr = [::1]:63300 req_id = 9840125c-6179-4fe4-a553-2aa3313117b5
80392025-04-30T06:43:01.102ZDEBGtest_snapshot_basic (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 51090,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 24,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 97,\n 100,\n 109,\n 105,\n 110,\n 45,\n 107,\n 101,\n 101,\n 112,\n 101,\n 114,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:63300 req_id = 9840125c-6179-4fe4-a553-2aa3313117b5
80402025-04-30T06:43:01.102ZDEBGtest_snapshot_basic (store): query key key = _clickhouse-admin-keeper._tcp
80412025-04-30T06:43:01.103ZERROtest_snapshot_basic (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 51090, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } NXDOMAIN: no records found for name: "_clickhouse-admin-keeper._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:63300 req_id = 9840125c-6179-4fe4-a553-2aa3313117b5
80422025-04-30T06:43:01.103ZDEBGtest_snapshot_basic (BackgroundTasks): begin collection background_task = inventory_collection
80432025-04-30T06:43:01.103ZDEBGtest_snapshot_basic (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:41502
80442025-04-30T06:43:01.103ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/ignition
80452025-04-30T06:43:01.105ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "e6125071-4d57-4ec0-a90c-541dd2032993", "content-length": "835", "date": "Wed, 30 Apr 2025 06:43:00 GMT"} })
80462025-04-30T06:43:01.105ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/0
80472025-04-30T06:43:01.106ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
80482025-04-30T06:43:01.124ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
80492025-04-30T06:43:01.124ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
80502025-04-30T06:43:01.124ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {} }
80512025-04-30T06:43:01.125ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
80522025-04-30T06:43:01.129ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
80532025-04-30T06:43:01.129ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {} }
80542025-04-30T06:43:01.130ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
80552025-04-30T06:43:01.130ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "2b04dc70-130c-4cca-86c0-466901313ec4", "content-length": "734", "date": "Wed, 30 Apr 2025 06:43:00 GMT"} })
80562025-04-30T06:43:01.131ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/0/component/sp/caboose?firmware_slot=0
80572025-04-30T06:43:01.135ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "21ba6bd3-6fdf-4780-8ad4-fe1fb9330e08", "content-length": "111", "date": "Wed, 30 Apr 2025 06:43:00 GMT"} })
80582025-04-30T06:43:01.135ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/0/component/sp/caboose?firmware_slot=1
80592025-04-30T06:43:01.137ZWARNtest_snapshot_basic (results-sink-cluster): failed to insert some results into metric DB collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 error = Failed to check out connection to database: No backends found for this service file = oximeter/collector/src/results_sink.rs:92
80602025-04-30T06:43:01.138ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "ef3a95e4-2a3a-4467-84f4-99fe1fe294f4", "content-length": "111", "date": "Wed, 30 Apr 2025 06:43:00 GMT"} })
80612025-04-30T06:43:01.138ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/0/component/rot/caboose?firmware_slot=0
80622025-04-30T06:43:01.142ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "6582a5fa-1b69-40f2-afcf-aa8475d57c97", "content-length": "170", "date": "Wed, 30 Apr 2025 06:43:00 GMT"} })
80632025-04-30T06:43:01.143ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/0/component/rot/caboose?firmware_slot=1
80642025-04-30T06:43:01.148ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "33d83e38-3437-4be5-a70c-f39003e0ffa3", "content-length": "170", "date": "Wed, 30 Apr 2025 06:43:00 GMT"} })
80652025-04-30T06:43:01.148ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/0/component/stage0/caboose?firmware_slot=0
80662025-04-30T06:43:01.151ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "37ea8b51-03e9-44d8-8cb1-4cf1b7fa3fa6", "content-length": "179", "date": "Wed, 30 Apr 2025 06:43:00 GMT"} })
80672025-04-30T06:43:01.151ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/0/component/stage0/caboose?firmware_slot=1
80682025-04-30T06:43:01.154ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "54b5a504-51c6-4225-aa1e-e8a5e1e1f29a", "content-length": "179", "date": "Wed, 30 Apr 2025 06:43:00 GMT"} })
80692025-04-30T06:43:01.154ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/0/component/rot/cmpa
80702025-04-30T06:43:01.155ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "bef80e88-9623-4eda-8287-e5d8bf58f1fc", "content-length": "702", "date": "Wed, 30 Apr 2025 06:43:00 GMT"} })
80712025-04-30T06:43:01.155ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/0/component/rot/cfpa
80722025-04-30T06:43:01.158ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "0fd4c7b3-0786-452d-8e87-fbd9736397cd", "content-length": "727", "date": "Wed, 30 Apr 2025 06:43:00 GMT"} })
80732025-04-30T06:43:01.158ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/0/component/rot/cfpa
80742025-04-30T06:43:01.158ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "e91e98fe-6f4b-498b-9e22-b7492eeb948b", "content-length": "729", "date": "Wed, 30 Apr 2025 06:43:00 GMT"} })
80752025-04-30T06:43:01.158ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/0/component/rot/cfpa
80762025-04-30T06:43:01.162ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "8d4babb6-0df6-4dca-b0ac-b4bd3348b25e", "content-length": "728", "date": "Wed, 30 Apr 2025 06:43:00 GMT"} })
80772025-04-30T06:43:01.162ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/1
80782025-04-30T06:43:01.162ZDEBGtest_snapshot_basic: attempt to fetch latest inventory collection result = Ok("found")
80792025-04-30T06:43:01.162ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "a2e5ba25-70c3-4003-b6f0-b539c9f86296", "content-length": "734", "date": "Wed, 30 Apr 2025 06:43:00 GMT"} })
80802025-04-30T06:43:01.162ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/1/component/sp/caboose?firmware_slot=0
80812025-04-30T06:43:01.167ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "228cc923-9477-45b8-84d9-d105b400fc08", "content-length": "111", "date": "Wed, 30 Apr 2025 06:43:00 GMT"} })
80822025-04-30T06:43:01.170ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/1/component/sp/caboose?firmware_slot=1
80832025-04-30T06:43:01.175ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "b92d9d7e-dfca-4da8-980f-a94d218b74a2", "content-length": "111", "date": "Wed, 30 Apr 2025 06:43:00 GMT"} })
80842025-04-30T06:43:01.175ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/1/component/rot/caboose?firmware_slot=0
80852025-04-30T06:43:01.179ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "ce0837e4-cd12-45ee-b092-431332a4f188", "content-length": "170", "date": "Wed, 30 Apr 2025 06:43:00 GMT"} })
80862025-04-30T06:43:01.183ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/1/component/rot/caboose?firmware_slot=1
80872025-04-30T06:43:01.183ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "46a1ce2c-2d74-4d1f-b4b5-0370bc0d7e34", "content-length": "170", "date": "Wed, 30 Apr 2025 06:43:00 GMT"} })
80882025-04-30T06:43:01.183ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/1/component/stage0/caboose?firmware_slot=0
80892025-04-30T06:43:01.183ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "1b5ad000-570b-48df-bd48-9d41a6d1d019", "content-length": "179", "date": "Wed, 30 Apr 2025 06:43:00 GMT"} })
80902025-04-30T06:43:01.183ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/1/component/stage0/caboose?firmware_slot=1
80912025-04-30T06:43:01.187ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "6a51daf8-5b5b-4ea5-b8d2-5e0797ea3c0e", "content-length": "179", "date": "Wed, 30 Apr 2025 06:43:00 GMT"} })
80922025-04-30T06:43:01.187ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/1/component/rot/cmpa
80932025-04-30T06:43:01.188ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "a1f209cd-0e5b-40f8-aca8-a0fe900ab43f", "content-length": "702", "date": "Wed, 30 Apr 2025 06:43:00 GMT"} })
80942025-04-30T06:43:01.189ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/1/component/rot/cfpa
80952025-04-30T06:43:01.190ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "1dd87be4-9f60-4e14-889f-f4a960924aec", "content-length": "727", "date": "Wed, 30 Apr 2025 06:43:00 GMT"} })
80962025-04-30T06:43:01.190ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/1/component/rot/cfpa
80972025-04-30T06:43:01.193ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "3925906c-5971-4240-b46c-45cbbd5bfdb3", "content-length": "729", "date": "Wed, 30 Apr 2025 06:43:00 GMT"} })
80982025-04-30T06:43:01.194ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/1/component/rot/cfpa
80992025-04-30T06:43:01.195ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "463619ec-34a9-4aba-92c9-7e43f2d71644", "content-length": "728", "date": "Wed, 30 Apr 2025 06:43:00 GMT"} })
81002025-04-30T06:43:01.195ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/0
81012025-04-30T06:43:01.196ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "1c3f04d0-1d77-428d-a27e-785385a2f984", "content-length": "723", "date": "Wed, 30 Apr 2025 06:43:00 GMT"} })
81022025-04-30T06:43:01.197ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/0/component/sp/caboose?firmware_slot=0
81032025-04-30T06:43:01.201ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "a4152d19-46e4-4a10-a84f-ae8d19f89bdb", "content-length": "109", "date": "Wed, 30 Apr 2025 06:43:00 GMT"} })
81042025-04-30T06:43:01.201ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/0/component/sp/caboose?firmware_slot=1
81052025-04-30T06:43:01.205ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "c89b48dd-83a5-4752-b104-504eab5bda1e", "content-length": "109", "date": "Wed, 30 Apr 2025 06:43:00 GMT"} })
81062025-04-30T06:43:01.205ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/0/component/rot/caboose?firmware_slot=0
81072025-04-30T06:43:01.212ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "b74b1f09-2808-44b8-9467-9832d2f6031c", "content-length": "169", "date": "Wed, 30 Apr 2025 06:43:00 GMT"} })
81082025-04-30T06:43:01.213ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/0/component/rot/caboose?firmware_slot=1
81092025-04-30T06:43:01.216ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
81102025-04-30T06:43:01.216ZTRACtest_snapshot_basic (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
81112025-04-30T06:43:01.216ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
81122025-04-30T06:43:01.220ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {} }
81132025-04-30T06:43:01.220ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
81142025-04-30T06:43:01.220ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "7135b46e-45d8-44df-a070-3cd5e37cdc3d", "content-length": "169", "date": "Wed, 30 Apr 2025 06:43:00 GMT"} })
81152025-04-30T06:43:01.220ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/0/component/stage0/caboose?firmware_slot=0
81162025-04-30T06:43:01.220ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
81172025-04-30T06:43:01.228ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
81182025-04-30T06:43:01.228ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
81192025-04-30T06:43:01.228ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {} }
81202025-04-30T06:43:01.229ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
81212025-04-30T06:43:01.232ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
81222025-04-30T06:43:01.233ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {} }
81232025-04-30T06:43:01.233ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
81242025-04-30T06:43:01.234ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "22343255-bcf8-4105-83ad-d6d7458f75d6", "content-length": "178", "date": "Wed, 30 Apr 2025 06:43:00 GMT"} })
81252025-04-30T06:43:01.234ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/0/component/stage0/caboose?firmware_slot=1
81262025-04-30T06:43:01.237ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "7309fbed-753a-461f-b370-9f2263456ea8", "content-length": "178", "date": "Wed, 30 Apr 2025 06:43:00 GMT"} })
81272025-04-30T06:43:01.237ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/0/component/rot/cmpa
81282025-04-30T06:43:01.238ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "20f8de71-531d-4f56-a894-05075ce667a4", "content-length": "702", "date": "Wed, 30 Apr 2025 06:43:00 GMT"} })
81292025-04-30T06:43:01.238ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/0/component/rot/cfpa
81302025-04-30T06:43:01.248ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "6152df1a-2656-4c03-9374-1a8e133d0e10", "content-length": "727", "date": "Wed, 30 Apr 2025 06:43:00 GMT"} })
81312025-04-30T06:43:01.248ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/0/component/rot/cfpa
81322025-04-30T06:43:01.249ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "ab0e60a6-6170-4991-afd1-6f5665e14327", "content-length": "729", "date": "Wed, 30 Apr 2025 06:43:00 GMT"} })
81332025-04-30T06:43:01.249ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/0/component/rot/cfpa
81342025-04-30T06:43:01.250ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "0372fda0-41a6-4de3-a5e5-dec3ab1b810e", "content-length": "728", "date": "Wed, 30 Apr 2025 06:43:00 GMT"} })
81352025-04-30T06:43:01.250ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/1
81362025-04-30T06:43:01.251ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "bc06d7e1-033d-446c-b8e8-60d977fc6073", "content-length": "723", "date": "Wed, 30 Apr 2025 06:43:00 GMT"} })
81372025-04-30T06:43:01.253ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/1/component/sp/caboose?firmware_slot=0
81382025-04-30T06:43:01.255ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "0204276a-c795-41af-9281-c56970da6fa4", "content-length": "109", "date": "Wed, 30 Apr 2025 06:43:00 GMT"} })
81392025-04-30T06:43:01.255ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/1/component/sp/caboose?firmware_slot=1
81402025-04-30T06:43:01.259ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "cc97b018-e4a0-45f0-91b1-e5fb25401656", "content-length": "109", "date": "Wed, 30 Apr 2025 06:43:00 GMT"} })
81412025-04-30T06:43:01.259ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/1/component/rot/caboose?firmware_slot=0
81422025-04-30T06:43:01.265ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "4b783893-aa31-45ba-85ef-edb6dfa248a6", "content-length": "169", "date": "Wed, 30 Apr 2025 06:43:00 GMT"} })
81432025-04-30T06:43:01.265ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/1/component/rot/caboose?firmware_slot=1
81442025-04-30T06:43:01.266ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "4922a84c-7537-4db0-bb2d-5b476aaa0e7b", "content-length": "169", "date": "Wed, 30 Apr 2025 06:43:00 GMT"} })
81452025-04-30T06:43:01.267ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/1/component/stage0/caboose?firmware_slot=0
81462025-04-30T06:43:01.269ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "46770e46-fd8c-4ff5-8a58-d342834bf55b", "content-length": "178", "date": "Wed, 30 Apr 2025 06:43:00 GMT"} })
81472025-04-30T06:43:01.269ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/1/component/stage0/caboose?firmware_slot=1
81482025-04-30T06:43:01.273ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "eb81c500-6d48-47d0-82dd-33c379d77627", "content-length": "178", "date": "Wed, 30 Apr 2025 06:43:00 GMT"} })
81492025-04-30T06:43:01.274ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/1/component/rot/cmpa
81502025-04-30T06:43:01.274ZDEBGtest_snapshot_basic: attempt to fetch latest inventory collection result = Ok("found")
81512025-04-30T06:43:01.275ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "bfae40a8-2d9e-42ab-b6f0-795a5f068a17", "content-length": "702", "date": "Wed, 30 Apr 2025 06:43:00 GMT"} })
81522025-04-30T06:43:01.275ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/1/component/rot/cfpa
81532025-04-30T06:43:01.277ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "15a09c43-19dc-4388-8d36-551d1ffe4262", "content-length": "727", "date": "Wed, 30 Apr 2025 06:43:00 GMT"} })
81542025-04-30T06:43:01.277ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/1/component/rot/cfpa
81552025-04-30T06:43:01.278ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "4e413f63-baae-4fd9-9d28-bcb687fb9263", "content-length": "729", "date": "Wed, 30 Apr 2025 06:43:00 GMT"} })
81562025-04-30T06:43:01.280ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/1/component/rot/cfpa
81572025-04-30T06:43:01.280ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "b0d62110-7d33-4a74-a1e8-b27848786541", "content-length": "728", "date": "Wed, 30 Apr 2025 06:43:00 GMT"} })
81582025-04-30T06:43:01.281ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
81592025-04-30T06:43:01.281ZTRACtest_snapshot_basic (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
81602025-04-30T06:43:01.281ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
81612025-04-30T06:43:01.281ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
81622025-04-30T06:43:01.281ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
81632025-04-30T06:43:01.285ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
81642025-04-30T06:43:01.292ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
81652025-04-30T06:43:01.293ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
81662025-04-30T06:43:01.293ZTRACtest_snapshot_basic (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
81672025-04-30T06:43:01.293ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
81682025-04-30T06:43:01.293ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
81692025-04-30T06:43:01.293ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
81702025-04-30T06:43:01.298ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
81712025-04-30T06:43:01.302ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
81722025-04-30T06:43:01.302ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
81732025-04-30T06:43:01.302ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
81742025-04-30T06:43:01.302ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
81752025-04-30T06:43:01.370ZDEBGtest_snapshot_basic (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:41923
81762025-04-30T06:43:01.370ZDEBGtest_snapshot_basic (BackgroundTasks): client request SledAgent = http://[::1]:41923 background_task = inventory_collection body = None method = GET uri = http://[::1]:41923/inventory
81772025-04-30T06:43:01.371ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
81782025-04-30T06:43:01.371ZTRACtest_snapshot_basic (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
81792025-04-30T06:43:01.371ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
81802025-04-30T06:43:01.371ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {} }
81812025-04-30T06:43:01.372ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
81822025-04-30T06:43:01.372ZINFOtest_snapshot_basic (dropshot): accepted connection local_addr = [::1]:41923 remote_addr = [::1]:64025 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
81832025-04-30T06:43:01.373ZTRACtest_snapshot_basic (dropshot): incoming request local_addr = [::1]:41923 method = GET remote_addr = [::1]:64025 req_id = 2fa76d38-06a8-4cf0-abf6-67da5dc19b02 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
81842025-04-30T06:43:01.373ZINFOtest_snapshot_basic (dropshot): request completed latency_us = 466 local_addr = [::1]:41923 method = GET remote_addr = [::1]:64025 req_id = 2fa76d38-06a8-4cf0-abf6-67da5dc19b02 response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
81852025-04-30T06:43:01.374ZDEBGtest_snapshot_basic (BackgroundTasks): client response SledAgent = http://[::1]:41923 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:41923/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "2fa76d38-06a8-4cf0-abf6-67da5dc19b02", "content-length": "3141", "date": "Wed, 30 Apr 2025 06:43:00 GMT"} })
81862025-04-30T06:43:01.376ZDEBGtest_snapshot_basic (BackgroundTasks): begin collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
81872025-04-30T06:43:01.376ZDEBGtest_snapshot_basic (BackgroundTasks): end collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
81882025-04-30T06:43:01.376ZDEBGtest_snapshot_basic (BackgroundTasks): finished collection background_task = inventory_collection
81892025-04-30T06:43:01.379ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
81902025-04-30T06:43:01.379ZTRACtest_snapshot_basic (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
81912025-04-30T06:43:01.379ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
81922025-04-30T06:43:01.379ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
81932025-04-30T06:43:01.380ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
81942025-04-30T06:43:01.380ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
81952025-04-30T06:43:01.385ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
81962025-04-30T06:43:01.385ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
81972025-04-30T06:43:01.385ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {} }
81982025-04-30T06:43:01.385ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
81992025-04-30T06:43:01.390ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
82002025-04-30T06:43:01.394ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
82012025-04-30T06:43:01.399ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
82022025-04-30T06:43:01.399ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
82032025-04-30T06:43:01.401ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
82042025-04-30T06:43:01.403ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
82052025-04-30T06:43:01.403ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {} }
82062025-04-30T06:43:01.404ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
82072025-04-30T06:43:01.436ZWARNtest_snapshot_basic (datastore): Non-retryable transaction failure err = DatabaseError(UniqueViolation, "duplicate key value violates unique constraint \\"inv_caboose_pkey\\"") transaction = inventory_insert_collection
82082025-04-30T06:43:01.437ZWARNtest_snapshot_basic (BackgroundTasks): inventory collection failed background_task = inventory_collection error = failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint "inv_caboose_pkey"
82092025-04-30T06:43:01.437ZDEBGtest_snapshot_basic (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 414.323142ms iteration = 26 status = {"error":"failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint \\"inv_caboose_pkey\\""}
82102025-04-30T06:43:01.437ZDEBGtest_snapshot_basic (BackgroundTasks): activating background_task = inventory_collection iteration = 27 reason = Signaled
82112025-04-30T06:43:01.437ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
82122025-04-30T06:43:01.437ZTRACtest_snapshot_basic (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
82132025-04-30T06:43:01.437ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
82142025-04-30T06:43:01.437ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
82152025-04-30T06:43:01.438ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
82162025-04-30T06:43:01.440ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
82172025-04-30T06:43:01.443ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
82182025-04-30T06:43:01.443ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
82192025-04-30T06:43:01.443ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
82202025-04-30T06:43:01.443ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
82212025-04-30T06:43:01.446ZDEBGtest_snapshot_basic: attempt to fetch latest inventory collection result = Ok("found")
82222025-04-30T06:43:01.448ZDEBGtest_snapshot_basic (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = [(d1ad44e5-18dd-4193-a70e-bf9042f0e0e1, 0)]
82232025-04-30T06:43:01.448ZTRACtest_snapshot_basic (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
82242025-04-30T06:43:01.451ZTRACtest_snapshot_basic (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: 5c a3 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 \\............_mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:56175 req_id = 68db00e1-0a53-4096-a0c7-5b22298a4188
82252025-04-30T06:43:01.453ZDEBGtest_snapshot_basic (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 23715,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:56175 req_id = 68db00e1-0a53-4096-a0c7-5b22298a4188
82262025-04-30T06:43:01.453ZDEBGtest_snapshot_basic (store): query key key = _mgs._tcp
82272025-04-30T06:43:01.454ZDEBGtest_snapshot_basic (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
82282025-04-30T06:43:01.454ZDEBGtest_snapshot_basic (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:56175 query = LowerQuery { name: LowerName(Name("_mgs._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 41502, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = 68db00e1-0a53-4096-a0c7-5b22298a4188
82292025-04-30T06:43:01.455ZDEBGtest_snapshot_basic (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 41502, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 4246, tv_nsec: 254387275 } })
82302025-04-30T06:43:01.455ZTRACtest_snapshot_basic (DnsResolver): lookup_service_targets: looking up SRV target
82312025-04-30T06:43:01.455ZTRACtest_snapshot_basic (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: 2a 4a 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e *J..........-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:62484 req_id = 4c169530-63c6-48c1-adfb-860046137da4
82322025-04-30T06:43:01.456ZDEBGtest_snapshot_basic (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 10826,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:62484 req_id = 4c169530-63c6-48c1-adfb-860046137da4
82332025-04-30T06:43:01.456ZDEBGtest_snapshot_basic (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
82342025-04-30T06:43:01.456ZDEBGtest_snapshot_basic (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:62484 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = 4c169530-63c6-48c1-adfb-860046137da4
82352025-04-30T06:43:01.505ZTRACtest_snapshot_basic (DnsResolver): lookup_all_socket_v6 srv dns_name = _clickhouse-admin-keeper._tcp.control-plane.oxide.internal
82362025-04-30T06:43:01.505ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
82372025-04-30T06:43:01.506ZTRACtest_snapshot_basic (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
82382025-04-30T06:43:01.506ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
82392025-04-30T06:43:01.506ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {} }
82402025-04-30T06:43:01.511ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
82412025-04-30T06:43:01.512ZTRACtest_snapshot_basic (dns): buffer buffer = Length: 87 (0x57) bytes\n0000: 3f c6 01 00 00 01 00 00 00 00 00 01 18 5f 63 6c ?............_cl\n0010: 69 63 6b 68 6f 75 73 65 2d 61 64 6d 69 6e 2d 6b ickhouse-admin-k\n0020: 65 65 70 65 72 04 5f 74 63 70 0d 63 6f 6e 74 72 eeper._tcp.contr\n0030: 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 69 ol-plane.oxide.i\n0040: 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 04 nternal..!....).\n0050: d0 00 00 00 00 00 00 ....... kind = dns peer_addr = [::1]:62451 req_id = 44fef3de-ef30-4831-aa16-e2f248c2a4fb
82422025-04-30T06:43:01.512ZDEBGtest_snapshot_basic (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 16326,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 24,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 97,\n 100,\n 109,\n 105,\n 110,\n 45,\n 107,\n 101,\n 101,\n 112,\n 101,\n 114,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:62451 req_id = 44fef3de-ef30-4831-aa16-e2f248c2a4fb
82432025-04-30T06:43:01.512ZDEBGtest_snapshot_basic (store): query key key = _clickhouse-admin-keeper._tcp
82442025-04-30T06:43:01.512ZERROtest_snapshot_basic (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 16326, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } NXDOMAIN: no records found for name: "_clickhouse-admin-keeper._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:62451 req_id = 44fef3de-ef30-4831-aa16-e2f248c2a4fb
82452025-04-30T06:43:01.513ZDEBGtest_snapshot_basic (BackgroundTasks): begin collection background_task = inventory_collection
82462025-04-30T06:43:01.513ZDEBGtest_snapshot_basic (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:41502
82472025-04-30T06:43:01.513ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/ignition
82482025-04-30T06:43:01.514ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
82492025-04-30T06:43:01.520ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
82502025-04-30T06:43:01.520ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
82512025-04-30T06:43:01.520ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {} }
82522025-04-30T06:43:01.521ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
82532025-04-30T06:43:01.537ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
82542025-04-30T06:43:01.537ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {} }
82552025-04-30T06:43:01.539ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
82562025-04-30T06:43:01.539ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "c57298b9-c5cb-4c90-83c5-b1411c8b9858", "content-length": "835", "date": "Wed, 30 Apr 2025 06:43:00 GMT"} })
82572025-04-30T06:43:01.540ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/0
82582025-04-30T06:43:01.560ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "a391996c-0786-4fd2-bd31-62737a045f57", "content-length": "734", "date": "Wed, 30 Apr 2025 06:43:00 GMT"} })
82592025-04-30T06:43:01.560ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/0/component/sp/caboose?firmware_slot=0
82602025-04-30T06:43:01.577ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "e892add0-6287-413f-ab8d-003d64ffdcff", "content-length": "111", "date": "Wed, 30 Apr 2025 06:43:00 GMT"} })
82612025-04-30T06:43:01.577ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/0/component/sp/caboose?firmware_slot=1
82622025-04-30T06:43:01.592ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "6d884e0a-8893-47df-87ac-ba204d830420", "content-length": "111", "date": "Wed, 30 Apr 2025 06:43:00 GMT"} })
82632025-04-30T06:43:01.593ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/0/component/rot/caboose?firmware_slot=0
82642025-04-30T06:43:01.602ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "4e1c232a-e6ba-4910-ba74-636f1f6bfd60", "content-length": "170", "date": "Wed, 30 Apr 2025 06:43:00 GMT"} })
82652025-04-30T06:43:01.603ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/0/component/rot/caboose?firmware_slot=1
82662025-04-30T06:43:01.609ZDEBGtest_snapshot_basic: attempt to fetch latest inventory collection result = Ok("found")
82672025-04-30T06:43:01.619ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "bc0054a5-47a7-4509-b07e-a61f3ed45e46", "content-length": "170", "date": "Wed, 30 Apr 2025 06:43:00 GMT"} })
82682025-04-30T06:43:01.619ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/0/component/stage0/caboose?firmware_slot=0
82692025-04-30T06:43:01.624ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "08a65237-f961-475b-8b20-f7133e0039f7", "content-length": "179", "date": "Wed, 30 Apr 2025 06:43:00 GMT"} })
82702025-04-30T06:43:01.627ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/0/component/stage0/caboose?firmware_slot=1
82712025-04-30T06:43:01.632ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "188f7470-c55d-4b2e-80fa-04e758bedb00", "content-length": "179", "date": "Wed, 30 Apr 2025 06:43:00 GMT"} })
82722025-04-30T06:43:01.632ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/0/component/rot/cmpa
82732025-04-30T06:43:01.635ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "990f502e-9434-4559-9cc3-aa20e9afed15", "content-length": "702", "date": "Wed, 30 Apr 2025 06:43:00 GMT"} })
82742025-04-30T06:43:01.635ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/0/component/rot/cfpa
82752025-04-30T06:43:01.638ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "1d6d5573-b7f7-4ddb-a74a-daf53fd40659", "content-length": "727", "date": "Wed, 30 Apr 2025 06:43:00 GMT"} })
82762025-04-30T06:43:01.638ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/0/component/rot/cfpa
82772025-04-30T06:43:01.640ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "6cf55bff-3ab1-4af3-8a2d-553ed4cf139e", "content-length": "729", "date": "Wed, 30 Apr 2025 06:43:00 GMT"} })
82782025-04-30T06:43:01.641ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/0/component/rot/cfpa
82792025-04-30T06:43:01.642ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "417d5e55-8c3a-435c-b5a6-119fcfdc4320", "content-length": "728", "date": "Wed, 30 Apr 2025 06:43:00 GMT"} })
82802025-04-30T06:43:01.642ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/1
82812025-04-30T06:43:01.648ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "557e329e-01d2-48b1-bc4c-4cce3d8ab2b1", "content-length": "734", "date": "Wed, 30 Apr 2025 06:43:00 GMT"} })
82822025-04-30T06:43:01.648ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/1/component/sp/caboose?firmware_slot=0
82832025-04-30T06:43:01.655ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "00a268a3-78d1-4cb1-95c6-790d4abbc036", "content-length": "111", "date": "Wed, 30 Apr 2025 06:43:00 GMT"} })
82842025-04-30T06:43:01.655ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/1/component/sp/caboose?firmware_slot=1
82852025-04-30T06:43:01.658ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "d6216651-9564-4c38-90bd-9faa3f0b2c4a", "content-length": "111", "date": "Wed, 30 Apr 2025 06:43:01 GMT"} })
82862025-04-30T06:43:01.658ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/1/component/rot/caboose?firmware_slot=0
82872025-04-30T06:43:01.661ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
82882025-04-30T06:43:01.661ZTRACtest_snapshot_basic (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
82892025-04-30T06:43:01.661ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
82902025-04-30T06:43:01.661ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {} }
82912025-04-30T06:43:01.662ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
82922025-04-30T06:43:01.662ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "aadd495d-a7e7-453b-897b-2bb8bdc09c9e", "content-length": "170", "date": "Wed, 30 Apr 2025 06:43:01 GMT"} })
82932025-04-30T06:43:01.662ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/1/component/rot/caboose?firmware_slot=1
82942025-04-30T06:43:01.674ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
82952025-04-30T06:43:01.687ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
82962025-04-30T06:43:01.687ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
82972025-04-30T06:43:01.687ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {} }
82982025-04-30T06:43:01.688ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
82992025-04-30T06:43:01.690ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
83002025-04-30T06:43:01.691ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {} }
83012025-04-30T06:43:01.691ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
83022025-04-30T06:43:01.692ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "2876b080-2b51-4cb3-9863-6c5e0d8eafc3", "content-length": "170", "date": "Wed, 30 Apr 2025 06:43:01 GMT"} })
83032025-04-30T06:43:01.692ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/1/component/stage0/caboose?firmware_slot=0
83042025-04-30T06:43:01.695ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "420d6ea8-add4-446f-a6a3-8ed7af7bd9ea", "content-length": "179", "date": "Wed, 30 Apr 2025 06:43:01 GMT"} })
83052025-04-30T06:43:01.695ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/1/component/stage0/caboose?firmware_slot=1
83062025-04-30T06:43:01.697ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "35800f46-e733-419c-8567-19d51dab09b4", "content-length": "179", "date": "Wed, 30 Apr 2025 06:43:01 GMT"} })
83072025-04-30T06:43:01.697ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/1/component/rot/cmpa
83082025-04-30T06:43:01.698ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "adad67c8-959c-4677-9877-7d1ed830745f", "content-length": "702", "date": "Wed, 30 Apr 2025 06:43:01 GMT"} })
83092025-04-30T06:43:01.698ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/1/component/rot/cfpa
83102025-04-30T06:43:01.699ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "52911fc6-0b2b-48ec-bd23-7d418a7687e2", "content-length": "727", "date": "Wed, 30 Apr 2025 06:43:01 GMT"} })
83112025-04-30T06:43:01.700ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/1/component/rot/cfpa
83122025-04-30T06:43:01.701ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "253777ac-bf7e-4c84-a731-b122c2626390", "content-length": "729", "date": "Wed, 30 Apr 2025 06:43:01 GMT"} })
83132025-04-30T06:43:01.701ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/1/component/rot/cfpa
83142025-04-30T06:43:01.702ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "8f52aadd-4856-496c-843d-4bec30ae7e07", "content-length": "728", "date": "Wed, 30 Apr 2025 06:43:01 GMT"} })
83152025-04-30T06:43:01.702ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/0
83162025-04-30T06:43:01.704ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "9e96e4d4-d89e-4a48-9fcc-d18bb1f6e3e0", "content-length": "723", "date": "Wed, 30 Apr 2025 06:43:01 GMT"} })
83172025-04-30T06:43:01.704ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/0/component/sp/caboose?firmware_slot=0
83182025-04-30T06:43:01.707ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "dcd34756-00a1-4c46-9c5f-5469ff99ed8d", "content-length": "109", "date": "Wed, 30 Apr 2025 06:43:01 GMT"} })
83192025-04-30T06:43:01.707ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/0/component/sp/caboose?firmware_slot=1
83202025-04-30T06:43:01.710ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "b0a00955-90c4-43f0-887b-bc01c278e10d", "content-length": "109", "date": "Wed, 30 Apr 2025 06:43:01 GMT"} })
83212025-04-30T06:43:01.710ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/0/component/rot/caboose?firmware_slot=0
83222025-04-30T06:43:01.712ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "2c6937a9-a8cf-4a50-ad1a-9463a0dee91c", "content-length": "169", "date": "Wed, 30 Apr 2025 06:43:01 GMT"} })
83232025-04-30T06:43:01.712ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/0/component/rot/caboose?firmware_slot=1
83242025-04-30T06:43:01.715ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "d30ef523-d93b-4325-91b9-ba69adc805e1", "content-length": "169", "date": "Wed, 30 Apr 2025 06:43:01 GMT"} })
83252025-04-30T06:43:01.715ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/0/component/stage0/caboose?firmware_slot=0
83262025-04-30T06:43:01.718ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "4e5a031f-b9e5-4831-876c-4895a6607b31", "content-length": "178", "date": "Wed, 30 Apr 2025 06:43:01 GMT"} })
83272025-04-30T06:43:01.718ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/0/component/stage0/caboose?firmware_slot=1
83282025-04-30T06:43:01.720ZINFOtest_snapshot_basic (oximeter-agent): refreshing list of producers from Nexus collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 file = oximeter/collector/src/agent.rs:562
83292025-04-30T06:43:01.721ZDEBGtest_snapshot_basic (oximeter): client request body = None method = GET uri = http://127.0.0.1:61468/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
83302025-04-30T06:43:01.721ZDEBGtest_snapshot_basic: attempt to fetch latest inventory collection result = Ok("found")
83312025-04-30T06:43:01.722ZTRACtest_snapshot_basic (dropshot_internal): incoming request local_addr = 127.0.0.1:61468 method = GET remote_addr = 127.0.0.1:64332 req_id = b4d1c0b9-9fe7-4c8d-8715-cdb6ea98d223 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
83322025-04-30T06:43:01.722ZTRACtest_snapshot_basic (dropshot_internal): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:61468 method = GET remote_addr = 127.0.0.1:64332 req_id = b4d1c0b9-9fe7-4c8d-8715-cdb6ea98d223 resource = Database uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
83332025-04-30T06:43:01.722ZDEBGtest_snapshot_basic (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:61468 method = GET remote_addr = 127.0.0.1:64332 req_id = b4d1c0b9-9fe7-4c8d-8715-cdb6ea98d223 roles = RoleSet { roles: {} } uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
83342025-04-30T06:43:01.723ZDEBGtest_snapshot_basic (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:61468 method = GET remote_addr = 127.0.0.1:64332 req_id = b4d1c0b9-9fe7-4c8d-8715-cdb6ea98d223 resource = Database result = Ok(()) uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
83352025-04-30T06:43:01.727ZINFOtest_snapshot_basic (dropshot_internal): request completed latency_us = 4614 local_addr = 127.0.0.1:61468 method = GET remote_addr = 127.0.0.1:64332 req_id = b4d1c0b9-9fe7-4c8d-8715-cdb6ea98d223 response_code = 200 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
83362025-04-30T06:43:01.727ZDEBGtest_snapshot_basic (oximeter): client response result = Ok(Response { url: "http://127.0.0.1:61468/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending", status: 200, headers: {"content-type": "application/json", "x-request-id": "b4d1c0b9-9fe7-4c8d-8715-cdb6ea98d223", "content-length": "405", "date": "Wed, 30 Apr 2025 06:43:01 GMT"} })
83372025-04-30T06:43:01.727ZDEBGtest_snapshot_basic (oximeter): client request body = None method = GET uri = http://127.0.0.1:61468/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
83382025-04-30T06:43:01.727ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "684fc34d-f617-48b9-a849-2bdfcfefd1f4", "content-length": "178", "date": "Wed, 30 Apr 2025 06:43:01 GMT"} })
83392025-04-30T06:43:01.727ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/0/component/rot/cmpa
83402025-04-30T06:43:01.728ZTRACtest_snapshot_basic (dropshot_internal): incoming request local_addr = 127.0.0.1:61468 method = GET remote_addr = 127.0.0.1:64332 req_id = ec459157-5f03-42a4-a1cc-d41ba8e320f8 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
83412025-04-30T06:43:01.730ZTRACtest_snapshot_basic (dropshot_internal): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:61468 method = GET remote_addr = 127.0.0.1:64332 req_id = ec459157-5f03-42a4-a1cc-d41ba8e320f8 resource = Database uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
83422025-04-30T06:43:01.730ZDEBGtest_snapshot_basic (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:61468 method = GET remote_addr = 127.0.0.1:64332 req_id = ec459157-5f03-42a4-a1cc-d41ba8e320f8 roles = RoleSet { roles: {} } uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
83432025-04-30T06:43:01.730ZDEBGtest_snapshot_basic (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:61468 method = GET remote_addr = 127.0.0.1:64332 req_id = ec459157-5f03-42a4-a1cc-d41ba8e320f8 resource = Database result = Ok(()) uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
83442025-04-30T06:43:01.732ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "4f1b41e7-4e46-425f-890a-89127db547bd", "content-length": "702", "date": "Wed, 30 Apr 2025 06:43:01 GMT"} })
83452025-04-30T06:43:01.733ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/0/component/rot/cfpa
83462025-04-30T06:43:01.734ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "768e3956-a0b1-4f64-b6a4-834a1ce865c7", "content-length": "727", "date": "Wed, 30 Apr 2025 06:43:01 GMT"} })
83472025-04-30T06:43:01.734ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/0/component/rot/cfpa
83482025-04-30T06:43:01.736ZINFOtest_snapshot_basic (dropshot_internal): request completed latency_us = 7844 local_addr = 127.0.0.1:61468 method = GET remote_addr = 127.0.0.1:64332 req_id = ec459157-5f03-42a4-a1cc-d41ba8e320f8 response_code = 200 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
83492025-04-30T06:43:01.736ZDEBGtest_snapshot_basic (oximeter): client response result = Ok(Response { url: "http://127.0.0.1:61468/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D", status: 200, headers: {"content-type": "application/json", "x-request-id": "ec459157-5f03-42a4-a1cc-d41ba8e320f8", "content-length": "29", "date": "Wed, 30 Apr 2025 06:43:01 GMT"} })
83502025-04-30T06:43:01.736ZTRACtest_snapshot_basic (oximeter-agent): ignoring request to update existing metric producer, since the endpoint information is the same as the existing collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = 354a52d4-3078-4706-81ce-12c32f6bbeaa
83512025-04-30T06:43:01.736ZTRACtest_snapshot_basic (oximeter-agent): ignoring request to update existing metric producer, since the endpoint information is the same as the existing collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
83522025-04-30T06:43:01.736ZINFOtest_snapshot_basic (oximeter-agent): refreshed list of producers from Nexus collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 file = oximeter/collector/src/agent.rs:637 n_current_tasks = 2 n_pruned_tasks = 0
83532025-04-30T06:43:01.737ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "f3275dec-a3bd-41a2-97f3-5d6eaf922084", "content-length": "729", "date": "Wed, 30 Apr 2025 06:43:01 GMT"} })
83542025-04-30T06:43:01.738ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/0/component/rot/cfpa
83552025-04-30T06:43:01.739ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "87f21715-c076-4640-a0e7-63e1155d98bc", "content-length": "728", "date": "Wed, 30 Apr 2025 06:43:01 GMT"} })
83562025-04-30T06:43:01.739ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/1
83572025-04-30T06:43:01.740ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "4c7bc0a2-8097-42d9-b6ab-dd79d4b2b883", "content-length": "723", "date": "Wed, 30 Apr 2025 06:43:01 GMT"} })
83582025-04-30T06:43:01.741ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/1/component/sp/caboose?firmware_slot=0
83592025-04-30T06:43:01.743ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "dccd6699-de13-41ff-b727-e41b89b155d2", "content-length": "109", "date": "Wed, 30 Apr 2025 06:43:01 GMT"} })
83602025-04-30T06:43:01.744ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/1/component/sp/caboose?firmware_slot=1
83612025-04-30T06:43:01.745ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "96b60294-4ae1-4c94-9a2e-a312a8d69a7c", "content-length": "109", "date": "Wed, 30 Apr 2025 06:43:01 GMT"} })
83622025-04-30T06:43:01.746ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/1/component/rot/caboose?firmware_slot=0
83632025-04-30T06:43:01.748ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "2a9b4049-3f33-4a68-91f9-48a6808a6bed", "content-length": "169", "date": "Wed, 30 Apr 2025 06:43:01 GMT"} })
83642025-04-30T06:43:01.749ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/1/component/rot/caboose?firmware_slot=1
83652025-04-30T06:43:01.754ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "a5d0b628-caaf-4998-ad56-90dac9a42137", "content-length": "169", "date": "Wed, 30 Apr 2025 06:43:01 GMT"} })
83662025-04-30T06:43:01.755ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/1/component/stage0/caboose?firmware_slot=0
83672025-04-30T06:43:01.759ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "f15871d8-9ae9-49ae-9a72-621a4770162a", "content-length": "178", "date": "Wed, 30 Apr 2025 06:43:01 GMT"} })
83682025-04-30T06:43:01.760ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/1/component/stage0/caboose?firmware_slot=1
83692025-04-30T06:43:01.764ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "3cdc5efd-92a3-4091-8f16-597a13c65a35", "content-length": "178", "date": "Wed, 30 Apr 2025 06:43:01 GMT"} })
83702025-04-30T06:43:01.765ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/1/component/rot/cmpa
83712025-04-30T06:43:01.766ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "83d9eb24-59b3-427d-95f1-5d641ba9588d", "content-length": "702", "date": "Wed, 30 Apr 2025 06:43:01 GMT"} })
83722025-04-30T06:43:01.766ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/1/component/rot/cfpa
83732025-04-30T06:43:01.767ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "f23daed3-8b13-49b4-9725-5d2beec5a404", "content-length": "727", "date": "Wed, 30 Apr 2025 06:43:01 GMT"} })
83742025-04-30T06:43:01.767ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/1/component/rot/cfpa
83752025-04-30T06:43:01.768ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "0c76300f-78a2-46cb-be51-3e5aa1ba1599", "content-length": "729", "date": "Wed, 30 Apr 2025 06:43:01 GMT"} })
83762025-04-30T06:43:01.768ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/1/component/rot/cfpa
83772025-04-30T06:43:01.769ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "eae37d31-41e2-40b1-8382-f6ad24653c60", "content-length": "728", "date": "Wed, 30 Apr 2025 06:43:01 GMT"} })
83782025-04-30T06:43:01.770ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
83792025-04-30T06:43:01.770ZTRACtest_snapshot_basic (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
83802025-04-30T06:43:01.770ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
83812025-04-30T06:43:01.770ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
83822025-04-30T06:43:01.772ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
83832025-04-30T06:43:01.773ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
83842025-04-30T06:43:01.773ZTRACtest_snapshot_basic (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
83852025-04-30T06:43:01.773ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
83862025-04-30T06:43:01.773ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {} }
83872025-04-30T06:43:01.776ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
83882025-04-30T06:43:01.779ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
83892025-04-30T06:43:01.784ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
83902025-04-30T06:43:01.784ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
83912025-04-30T06:43:01.784ZTRACtest_snapshot_basic (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
83922025-04-30T06:43:01.784ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
83932025-04-30T06:43:01.784ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
83942025-04-30T06:43:01.787ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
83952025-04-30T06:43:01.787ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
83962025-04-30T06:43:01.795ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
83972025-04-30T06:43:01.795ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
83982025-04-30T06:43:01.796ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {} }
83992025-04-30T06:43:01.796ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
84002025-04-30T06:43:01.798ZDEBGtest_snapshot_basic (collection-task): sent timer-based collection request to the collection task collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
84012025-04-30T06:43:01.798ZDEBGtest_snapshot_basic (collection-task): collecting from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
84022025-04-30T06:43:01.799ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
84032025-04-30T06:43:01.803ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
84042025-04-30T06:43:01.803ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
84052025-04-30T06:43:01.803ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
84062025-04-30T06:43:01.804ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
84072025-04-30T06:43:01.806ZTRACtest_snapshot_basic (collection-task): sent collection request to producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
84082025-04-30T06:43:01.806ZDEBGtest_snapshot_basic (collection-task): collected results from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 n_results = 1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
84092025-04-30T06:43:01.807ZTRACtest_snapshot_basic (collection-task): forwarded results to main collection loop collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
84102025-04-30T06:43:01.814ZTRACtest_snapshot_basic (collection-task): top of inner collection loop, waiting for next request collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
84112025-04-30T06:43:01.814ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
84122025-04-30T06:43:01.814ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {} }
84132025-04-30T06:43:01.814ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
84142025-04-30T06:43:01.919ZDEBGtest_snapshot_basic (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:41923
84152025-04-30T06:43:01.919ZDEBGtest_snapshot_basic (BackgroundTasks): client request SledAgent = http://[::1]:41923 background_task = inventory_collection body = None method = GET uri = http://[::1]:41923/inventory
84162025-04-30T06:43:01.920ZINFOtest_snapshot_basic (dropshot): accepted connection local_addr = [::1]:41923 remote_addr = [::1]:61184 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
84172025-04-30T06:43:01.922ZTRACtest_snapshot_basic (dropshot): incoming request local_addr = [::1]:41923 method = GET remote_addr = [::1]:61184 req_id = 86570c6a-2d82-4f11-8f21-eabfa354ecf3 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
84182025-04-30T06:43:01.922ZINFOtest_snapshot_basic (dropshot): request completed latency_us = 638 local_addr = [::1]:41923 method = GET remote_addr = [::1]:61184 req_id = 86570c6a-2d82-4f11-8f21-eabfa354ecf3 response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
84192025-04-30T06:43:01.923ZDEBGtest_snapshot_basic (BackgroundTasks): client response SledAgent = http://[::1]:41923 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:41923/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "86570c6a-2d82-4f11-8f21-eabfa354ecf3", "content-length": "3141", "date": "Wed, 30 Apr 2025 06:43:01 GMT"} })
84202025-04-30T06:43:01.925ZDEBGtest_snapshot_basic (BackgroundTasks): begin collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
84212025-04-30T06:43:01.925ZDEBGtest_snapshot_basic (BackgroundTasks): end collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
84222025-04-30T06:43:01.925ZDEBGtest_snapshot_basic (BackgroundTasks): finished collection background_task = inventory_collection
84232025-04-30T06:43:01.926ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
84242025-04-30T06:43:01.927ZTRACtest_snapshot_basic (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
84252025-04-30T06:43:01.927ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
84262025-04-30T06:43:01.927ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
84272025-04-30T06:43:01.927ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
84282025-04-30T06:43:01.929ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
84292025-04-30T06:43:01.935ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
84302025-04-30T06:43:01.935ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
84312025-04-30T06:43:01.935ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
84322025-04-30T06:43:01.937ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
84332025-04-30T06:43:01.969ZWARNtest_snapshot_basic (datastore): Non-retryable transaction failure err = DatabaseError(UniqueViolation, "duplicate key value violates unique constraint \\"inv_caboose_pkey\\"") transaction = inventory_insert_collection
84342025-04-30T06:43:01.969ZWARNtest_snapshot_basic (BackgroundTasks): inventory collection failed background_task = inventory_collection error = failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint "inv_caboose_pkey"
84352025-04-30T06:43:01.969ZDEBGtest_snapshot_basic (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 532.352397ms iteration = 27 status = {"error":"failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint \\"inv_caboose_pkey\\""}
84362025-04-30T06:43:01.969ZDEBGtest_snapshot_basic (BackgroundTasks): activating background_task = inventory_collection iteration = 28 reason = Signaled
84372025-04-30T06:43:01.969ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
84382025-04-30T06:43:01.969ZTRACtest_snapshot_basic (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
84392025-04-30T06:43:01.969ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
84402025-04-30T06:43:01.969ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
84412025-04-30T06:43:01.970ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
84422025-04-30T06:43:01.971ZDEBGtest_snapshot_basic: attempt to fetch latest inventory collection result = Ok("found")
84432025-04-30T06:43:01.973ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
84442025-04-30T06:43:01.988ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
84452025-04-30T06:43:01.988ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
84462025-04-30T06:43:01.988ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
84472025-04-30T06:43:01.990ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
84482025-04-30T06:43:02.003ZDEBGtest_snapshot_basic (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = [(d1ad44e5-18dd-4193-a70e-bf9042f0e0e1, 0)]
84492025-04-30T06:43:02.003ZTRACtest_snapshot_basic (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
84502025-04-30T06:43:02.005ZTRACtest_snapshot_basic (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: d6 f9 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 ............._mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:56068 req_id = 67a58896-5ec0-4763-8fd1-dddc08a90614
84512025-04-30T06:43:02.005ZDEBGtest_snapshot_basic (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 55033,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:56068 req_id = 67a58896-5ec0-4763-8fd1-dddc08a90614
84522025-04-30T06:43:02.005ZDEBGtest_snapshot_basic (store): query key key = _mgs._tcp
84532025-04-30T06:43:02.006ZDEBGtest_snapshot_basic (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
84542025-04-30T06:43:02.006ZDEBGtest_snapshot_basic (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:56068 query = LowerQuery { name: LowerName(Name("_mgs._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 41502, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = 67a58896-5ec0-4763-8fd1-dddc08a90614
84552025-04-30T06:43:02.006ZDEBGtest_snapshot_basic (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 41502, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 4246, tv_nsec: 805721882 } })
84562025-04-30T06:43:02.007ZTRACtest_snapshot_basic (DnsResolver): lookup_service_targets: looking up SRV target
84572025-04-30T06:43:02.007ZTRACtest_snapshot_basic (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: d9 52 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e .R..........-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:53443 req_id = 1eb5516c-9d27-4113-9cfd-8d476712b7f3
84582025-04-30T06:43:02.007ZDEBGtest_snapshot_basic (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 55634,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:53443 req_id = 1eb5516c-9d27-4113-9cfd-8d476712b7f3
84592025-04-30T06:43:02.007ZDEBGtest_snapshot_basic (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
84602025-04-30T06:43:02.008ZDEBGtest_snapshot_basic (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:53443 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = 1eb5516c-9d27-4113-9cfd-8d476712b7f3
84612025-04-30T06:43:02.068ZTRACtest_snapshot_basic (DnsResolver): lookup_all_socket_v6 srv dns_name = _clickhouse-admin-keeper._tcp.control-plane.oxide.internal
84622025-04-30T06:43:02.069ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
84632025-04-30T06:43:02.069ZTRACtest_snapshot_basic (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
84642025-04-30T06:43:02.069ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
84652025-04-30T06:43:02.069ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {} }
84662025-04-30T06:43:02.070ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
84672025-04-30T06:43:02.071ZDEBGtest_snapshot_basic (results-sink): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
84682025-04-30T06:43:02.071ZDEBGtest_snapshot_basic (results-sink-cluster): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
84692025-04-30T06:43:02.072ZDEBGtest_snapshot_basic (clickhouse-client): unrolling 1 total samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 6ad06c6b-9a6e-4ed1-b3f4-5d2e9da3f8a7
84702025-04-30T06:43:02.073ZTRACtest_snapshot_basic (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 6ad06c6b-9a6e-4ed1-b3f4-5d2e9da3f8a7 n_columns = 4 n_rows = 2 sql = INSERT INTO oximeter.fields_string FORMAT Native
84712025-04-30T06:43:02.075ZTRACtest_snapshot_basic (dns): buffer buffer = Length: 87 (0x57) bytes\n0000: a1 13 01 00 00 01 00 00 00 00 00 01 18 5f 63 6c ............._cl\n0010: 69 63 6b 68 6f 75 73 65 2d 61 64 6d 69 6e 2d 6b ickhouse-admin-k\n0020: 65 65 70 65 72 04 5f 74 63 70 0d 63 6f 6e 74 72 eeper._tcp.contr\n0030: 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 69 ol-plane.oxide.i\n0040: 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 04 nternal..!....).\n0050: d0 00 00 00 00 00 00 ....... kind = dns peer_addr = [::1]:50811 req_id = 359ded0c-16a0-4e39-99eb-524d5405001a
84722025-04-30T06:43:02.075ZDEBGtest_snapshot_basic (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 41235,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 24,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 97,\n 100,\n 109,\n 105,\n 110,\n 45,\n 107,\n 101,\n 101,\n 112,\n 101,\n 114,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:50811 req_id = 359ded0c-16a0-4e39-99eb-524d5405001a
84732025-04-30T06:43:02.076ZDEBGtest_snapshot_basic (store): query key key = _clickhouse-admin-keeper._tcp
84742025-04-30T06:43:02.076ZERROtest_snapshot_basic (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 41235, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } NXDOMAIN: no records found for name: "_clickhouse-admin-keeper._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:50811 req_id = 359ded0c-16a0-4e39-99eb-524d5405001a
84752025-04-30T06:43:02.076ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
84762025-04-30T06:43:02.082ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
84772025-04-30T06:43:02.082ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
84782025-04-30T06:43:02.082ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {} }
84792025-04-30T06:43:02.083ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
84802025-04-30T06:43:02.084ZDEBGtest_snapshot_basic (BackgroundTasks): begin collection background_task = inventory_collection
84812025-04-30T06:43:02.084ZDEBGtest_snapshot_basic (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:41502
84822025-04-30T06:43:02.085ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/ignition
84832025-04-30T06:43:02.087ZDEBGtest_snapshot_basic (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 6ad06c6b-9a6e-4ed1-b3f4-5d2e9da3f8a7 n_rows = 2 table_name = fields_string
84842025-04-30T06:43:02.087ZTRACtest_snapshot_basic (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 6ad06c6b-9a6e-4ed1-b3f4-5d2e9da3f8a7 n_columns = 4 n_rows = 1 sql = INSERT INTO oximeter.measurements_i64 FORMAT Native
84852025-04-30T06:43:02.089ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
84862025-04-30T06:43:02.090ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {} }
84872025-04-30T06:43:02.090ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
84882025-04-30T06:43:02.092ZDEBGtest_snapshot_basic (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 6ad06c6b-9a6e-4ed1-b3f4-5d2e9da3f8a7 n_rows = 1 table_name = measurements_i64
84892025-04-30T06:43:02.092ZTRACtest_snapshot_basic (results-sink): successfully inserted samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
84902025-04-30T06:43:02.092ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "fb0f2996-9e77-41d4-b844-fa29c1a7490c", "content-length": "835", "date": "Wed, 30 Apr 2025 06:43:01 GMT"} })
84912025-04-30T06:43:02.095ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/0
84922025-04-30T06:43:02.095ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "32612cd2-36f0-4d99-814c-baa0f5f8ffab", "content-length": "734", "date": "Wed, 30 Apr 2025 06:43:01 GMT"} })
84932025-04-30T06:43:02.096ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/0/component/sp/caboose?firmware_slot=0
84942025-04-30T06:43:02.099ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "10563909-bf76-4c77-bcd9-b96dee4b2674", "content-length": "111", "date": "Wed, 30 Apr 2025 06:43:01 GMT"} })
84952025-04-30T06:43:02.099ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/0/component/sp/caboose?firmware_slot=1
84962025-04-30T06:43:02.100ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "8a032ec3-8342-464a-b9aa-ea775d2bc2c3", "content-length": "111", "date": "Wed, 30 Apr 2025 06:43:01 GMT"} })
84972025-04-30T06:43:02.100ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/0/component/rot/caboose?firmware_slot=0
84982025-04-30T06:43:02.101ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "9015cb27-244c-49a4-8178-376c5d2b5dfb", "content-length": "170", "date": "Wed, 30 Apr 2025 06:43:01 GMT"} })
84992025-04-30T06:43:02.101ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/0/component/rot/caboose?firmware_slot=1
85002025-04-30T06:43:02.103ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "70d9f34d-c823-40fc-ba3f-627aa58cb3ae", "content-length": "170", "date": "Wed, 30 Apr 2025 06:43:01 GMT"} })
85012025-04-30T06:43:02.103ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/0/component/stage0/caboose?firmware_slot=0
85022025-04-30T06:43:02.105ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "0844a7a1-6c95-4b18-ab77-8cfdfd38c83f", "content-length": "179", "date": "Wed, 30 Apr 2025 06:43:01 GMT"} })
85032025-04-30T06:43:02.105ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/0/component/stage0/caboose?firmware_slot=1
85042025-04-30T06:43:02.107ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "81909d9e-7189-4028-8df7-10fbbaa35348", "content-length": "179", "date": "Wed, 30 Apr 2025 06:43:01 GMT"} })
85052025-04-30T06:43:02.107ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/0/component/rot/cmpa
85062025-04-30T06:43:02.109ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "b8908429-f4ec-45f1-b87e-fff2e9c56022", "content-length": "702", "date": "Wed, 30 Apr 2025 06:43:01 GMT"} })
85072025-04-30T06:43:02.110ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/0/component/rot/cfpa
85082025-04-30T06:43:02.110ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "e5bef93f-fb41-40df-884a-6e58607b2d49", "content-length": "727", "date": "Wed, 30 Apr 2025 06:43:01 GMT"} })
85092025-04-30T06:43:02.110ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/0/component/rot/cfpa
85102025-04-30T06:43:02.111ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "873a8cd3-1a6b-4ddf-8509-9512f32a7def", "content-length": "729", "date": "Wed, 30 Apr 2025 06:43:01 GMT"} })
85112025-04-30T06:43:02.111ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/0/component/rot/cfpa
85122025-04-30T06:43:02.113ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "af4730b2-8c8a-42b7-acfb-409badbda91c", "content-length": "728", "date": "Wed, 30 Apr 2025 06:43:01 GMT"} })
85132025-04-30T06:43:02.113ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/1
85142025-04-30T06:43:02.114ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "24170b21-cdde-4d15-929e-596007314b37", "content-length": "734", "date": "Wed, 30 Apr 2025 06:43:01 GMT"} })
85152025-04-30T06:43:02.114ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/1/component/sp/caboose?firmware_slot=0
85162025-04-30T06:43:02.117ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "e6be4273-484c-4784-84cd-1dd99d665826", "content-length": "111", "date": "Wed, 30 Apr 2025 06:43:01 GMT"} })
85172025-04-30T06:43:02.118ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/1/component/sp/caboose?firmware_slot=1
85182025-04-30T06:43:02.121ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "1dbbb6fb-ac10-43dd-a5b3-ea0366488b2d", "content-length": "111", "date": "Wed, 30 Apr 2025 06:43:01 GMT"} })
85192025-04-30T06:43:02.121ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/1/component/rot/caboose?firmware_slot=0
85202025-04-30T06:43:02.121ZDEBGtest_snapshot_basic: attempt to fetch latest inventory collection result = Ok("found")
85212025-04-30T06:43:02.123ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "a0953300-9ab6-4a9a-a1d6-8656fb7ca9dc", "content-length": "170", "date": "Wed, 30 Apr 2025 06:43:01 GMT"} })
85222025-04-30T06:43:02.124ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/1/component/rot/caboose?firmware_slot=1
85232025-04-30T06:43:02.125ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "c3bdfc81-5141-4b75-a281-4be2725a176b", "content-length": "170", "date": "Wed, 30 Apr 2025 06:43:01 GMT"} })
85242025-04-30T06:43:02.125ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/1/component/stage0/caboose?firmware_slot=0
85252025-04-30T06:43:02.128ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "6f3877f3-b2b8-4bf6-9370-68e012dae7c6", "content-length": "179", "date": "Wed, 30 Apr 2025 06:43:01 GMT"} })
85262025-04-30T06:43:02.128ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/1/component/stage0/caboose?firmware_slot=1
85272025-04-30T06:43:02.130ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "20292e72-dc7d-4d3a-907d-dd4bee9576a6", "content-length": "179", "date": "Wed, 30 Apr 2025 06:43:01 GMT"} })
85282025-04-30T06:43:02.130ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/1/component/rot/cmpa
85292025-04-30T06:43:02.130ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "8fd939c8-0e3b-489b-8714-45412a228dcb", "content-length": "702", "date": "Wed, 30 Apr 2025 06:43:01 GMT"} })
85302025-04-30T06:43:02.130ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/1/component/rot/cfpa
85312025-04-30T06:43:02.131ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "4538ab4d-f6c6-4795-b243-6f8ed47c975e", "content-length": "727", "date": "Wed, 30 Apr 2025 06:43:01 GMT"} })
85322025-04-30T06:43:02.131ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/1/component/rot/cfpa
85332025-04-30T06:43:02.132ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "1d5268fa-9000-4a75-b798-ce904c5baddb", "content-length": "729", "date": "Wed, 30 Apr 2025 06:43:01 GMT"} })
85342025-04-30T06:43:02.132ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/1/component/rot/cfpa
85352025-04-30T06:43:02.132ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "f8e25c98-70c2-4ed5-a762-eb8d5e074cce", "content-length": "728", "date": "Wed, 30 Apr 2025 06:43:01 GMT"} })
85362025-04-30T06:43:02.133ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/0
85372025-04-30T06:43:02.134ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "88b0f5f2-e4e9-4a78-8767-3a936e5dc53e", "content-length": "723", "date": "Wed, 30 Apr 2025 06:43:01 GMT"} })
85382025-04-30T06:43:02.134ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/0/component/sp/caboose?firmware_slot=0
85392025-04-30T06:43:02.135ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "262d5f02-175f-418f-b24b-833da29df984", "content-length": "109", "date": "Wed, 30 Apr 2025 06:43:01 GMT"} })
85402025-04-30T06:43:02.135ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/0/component/sp/caboose?firmware_slot=1
85412025-04-30T06:43:02.137ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "c7f4fd57-93f5-4e0f-b72c-33296f51b941", "content-length": "109", "date": "Wed, 30 Apr 2025 06:43:01 GMT"} })
85422025-04-30T06:43:02.137ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/0/component/rot/caboose?firmware_slot=0
85432025-04-30T06:43:02.139ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "3efdcee5-7ba8-422c-a868-251858cb8624", "content-length": "169", "date": "Wed, 30 Apr 2025 06:43:01 GMT"} })
85442025-04-30T06:43:02.139ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/0/component/rot/caboose?firmware_slot=1
85452025-04-30T06:43:02.140ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "f40461a9-6b17-450d-828e-5799c10ec381", "content-length": "169", "date": "Wed, 30 Apr 2025 06:43:01 GMT"} })
85462025-04-30T06:43:02.141ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/0/component/stage0/caboose?firmware_slot=0
85472025-04-30T06:43:02.142ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "76acc187-d3ac-4293-b344-892f2a856714", "content-length": "178", "date": "Wed, 30 Apr 2025 06:43:01 GMT"} })
85482025-04-30T06:43:02.142ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/0/component/stage0/caboose?firmware_slot=1
85492025-04-30T06:43:02.144ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "00205598-cd81-45c3-9e05-6ffa67bcc1a2", "content-length": "178", "date": "Wed, 30 Apr 2025 06:43:01 GMT"} })
85502025-04-30T06:43:02.144ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/0/component/rot/cmpa
85512025-04-30T06:43:02.145ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "b3fbc515-87a0-45d2-b79a-65d68ea76145", "content-length": "702", "date": "Wed, 30 Apr 2025 06:43:01 GMT"} })
85522025-04-30T06:43:02.145ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/0/component/rot/cfpa
85532025-04-30T06:43:02.146ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "7f0b7b21-cd8e-4e92-9913-08c30f3f89e6", "content-length": "727", "date": "Wed, 30 Apr 2025 06:43:01 GMT"} })
85542025-04-30T06:43:02.146ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/0/component/rot/cfpa
85552025-04-30T06:43:02.147ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "f10722da-8300-4c2e-be27-1eb4f2a6ec2f", "content-length": "729", "date": "Wed, 30 Apr 2025 06:43:01 GMT"} })
85562025-04-30T06:43:02.147ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/0/component/rot/cfpa
85572025-04-30T06:43:02.148ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "498008cd-3a1d-4c8e-a08a-272bc450c7e0", "content-length": "728", "date": "Wed, 30 Apr 2025 06:43:01 GMT"} })
85582025-04-30T06:43:02.149ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/1
85592025-04-30T06:43:02.149ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "5f9523d0-7b81-4fd2-b99e-e2b5d4f930b8", "content-length": "723", "date": "Wed, 30 Apr 2025 06:43:01 GMT"} })
85602025-04-30T06:43:02.150ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/1/component/sp/caboose?firmware_slot=0
85612025-04-30T06:43:02.151ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "09c1d84f-4c21-4734-8f4a-8451520add4a", "content-length": "109", "date": "Wed, 30 Apr 2025 06:43:01 GMT"} })
85622025-04-30T06:43:02.151ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/1/component/sp/caboose?firmware_slot=1
85632025-04-30T06:43:02.154ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "f7a328b5-abb0-4e54-a930-34e15f26e787", "content-length": "109", "date": "Wed, 30 Apr 2025 06:43:01 GMT"} })
85642025-04-30T06:43:02.154ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/1/component/rot/caboose?firmware_slot=0
85652025-04-30T06:43:02.157ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "7784840c-b49e-456f-90c3-07851a130a0a", "content-length": "169", "date": "Wed, 30 Apr 2025 06:43:01 GMT"} })
85662025-04-30T06:43:02.157ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/1/component/rot/caboose?firmware_slot=1
85672025-04-30T06:43:02.158ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "1f798dd4-0bdf-4f65-ab07-dc621d88982c", "content-length": "169", "date": "Wed, 30 Apr 2025 06:43:01 GMT"} })
85682025-04-30T06:43:02.158ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/1/component/stage0/caboose?firmware_slot=0
85692025-04-30T06:43:02.160ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "5ca7446a-90e8-43a4-92c9-163250a2dca0", "content-length": "178", "date": "Wed, 30 Apr 2025 06:43:01 GMT"} })
85702025-04-30T06:43:02.160ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/1/component/stage0/caboose?firmware_slot=1
85712025-04-30T06:43:02.162ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "161152c5-6b21-4303-8912-ca701c148593", "content-length": "178", "date": "Wed, 30 Apr 2025 06:43:01 GMT"} })
85722025-04-30T06:43:02.162ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/1/component/rot/cmpa
85732025-04-30T06:43:02.165ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "8bc15970-8595-46b8-8daf-1c99269e65ae", "content-length": "702", "date": "Wed, 30 Apr 2025 06:43:01 GMT"} })
85742025-04-30T06:43:02.165ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/1/component/rot/cfpa
85752025-04-30T06:43:02.167ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "5bd2085b-342a-4964-89f3-121207ace10e", "content-length": "727", "date": "Wed, 30 Apr 2025 06:43:01 GMT"} })
85762025-04-30T06:43:02.167ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/1/component/rot/cfpa
85772025-04-30T06:43:02.167ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "d594e20e-e0df-458b-ae08-a9106cc46a84", "content-length": "729", "date": "Wed, 30 Apr 2025 06:43:01 GMT"} })
85782025-04-30T06:43:02.167ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/1/component/rot/cfpa
85792025-04-30T06:43:02.169ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "95cc2488-a1c3-461a-8f48-9481aa85bd0a", "content-length": "728", "date": "Wed, 30 Apr 2025 06:43:01 GMT"} })
85802025-04-30T06:43:02.169ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
85812025-04-30T06:43:02.169ZTRACtest_snapshot_basic (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
85822025-04-30T06:43:02.169ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
85832025-04-30T06:43:02.169ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
85842025-04-30T06:43:02.170ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
85852025-04-30T06:43:02.183ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
85862025-04-30T06:43:02.183ZTRACtest_snapshot_basic (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
85872025-04-30T06:43:02.183ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
85882025-04-30T06:43:02.183ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {} }
85892025-04-30T06:43:02.183ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
85902025-04-30T06:43:02.183ZWARNtest_snapshot_basic (results-sink-cluster): failed to insert some results into metric DB collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 error = Failed to check out connection to database: No backends found for this service file = oximeter/collector/src/results_sink.rs:92
85912025-04-30T06:43:02.183ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
85922025-04-30T06:43:02.184ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
85932025-04-30T06:43:02.184ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
85942025-04-30T06:43:02.184ZTRACtest_snapshot_basic (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
85952025-04-30T06:43:02.184ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
85962025-04-30T06:43:02.184ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
85972025-04-30T06:43:02.184ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
85982025-04-30T06:43:02.188ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
85992025-04-30T06:43:02.195ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
86002025-04-30T06:43:02.195ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
86012025-04-30T06:43:02.195ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {} }
86022025-04-30T06:43:02.195ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
86032025-04-30T06:43:02.198ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
86042025-04-30T06:43:02.204ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
86052025-04-30T06:43:02.204ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
86062025-04-30T06:43:02.205ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
86072025-04-30T06:43:02.208ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
86082025-04-30T06:43:02.208ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
86092025-04-30T06:43:02.208ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {} }
86102025-04-30T06:43:02.210ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
86112025-04-30T06:43:02.314ZDEBGtest_snapshot_basic (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:41923
86122025-04-30T06:43:02.315ZDEBGtest_snapshot_basic (BackgroundTasks): client request SledAgent = http://[::1]:41923 background_task = inventory_collection body = None method = GET uri = http://[::1]:41923/inventory
86132025-04-30T06:43:02.316ZINFOtest_snapshot_basic (dropshot): accepted connection local_addr = [::1]:41923 remote_addr = [::1]:39514 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
86142025-04-30T06:43:02.317ZTRACtest_snapshot_basic (dropshot): incoming request local_addr = [::1]:41923 method = GET remote_addr = [::1]:39514 req_id = 7b3be608-8c13-4c8f-b686-c6d54b687faa sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
86152025-04-30T06:43:02.318ZINFOtest_snapshot_basic (dropshot): request completed latency_us = 1981 local_addr = [::1]:41923 method = GET remote_addr = [::1]:39514 req_id = 7b3be608-8c13-4c8f-b686-c6d54b687faa response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
86162025-04-30T06:43:02.319ZDEBGtest_snapshot_basic (BackgroundTasks): client response SledAgent = http://[::1]:41923 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:41923/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "7b3be608-8c13-4c8f-b686-c6d54b687faa", "content-length": "3141", "date": "Wed, 30 Apr 2025 06:43:01 GMT"} })
86172025-04-30T06:43:02.323ZDEBGtest_snapshot_basic (BackgroundTasks): begin collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
86182025-04-30T06:43:02.324ZDEBGtest_snapshot_basic (BackgroundTasks): end collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
86192025-04-30T06:43:02.324ZDEBGtest_snapshot_basic (BackgroundTasks): finished collection background_task = inventory_collection
86202025-04-30T06:43:02.326ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
86212025-04-30T06:43:02.326ZTRACtest_snapshot_basic (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
86222025-04-30T06:43:02.326ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
86232025-04-30T06:43:02.327ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
86242025-04-30T06:43:02.327ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
86252025-04-30T06:43:02.341ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
86262025-04-30T06:43:02.351ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
86272025-04-30T06:43:02.352ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
86282025-04-30T06:43:02.352ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
86292025-04-30T06:43:02.352ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
86302025-04-30T06:43:02.423ZDEBGtest_snapshot_basic: attempt to fetch latest inventory collection result = Ok("found")
86312025-04-30T06:43:02.427ZWARNtest_snapshot_basic (datastore): Non-retryable transaction failure err = DatabaseError(UniqueViolation, "duplicate key value violates unique constraint \\"inv_caboose_pkey\\"") transaction = inventory_insert_collection
86322025-04-30T06:43:02.427ZWARNtest_snapshot_basic (BackgroundTasks): inventory collection failed background_task = inventory_collection error = failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint "inv_caboose_pkey"
86332025-04-30T06:43:02.427ZDEBGtest_snapshot_basic (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 455.469203ms iteration = 28 status = {"error":"failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint \\"inv_caboose_pkey\\""}
86342025-04-30T06:43:02.430ZDEBGtest_snapshot_basic (BackgroundTasks): activating background_task = inventory_collection iteration = 29 reason = Signaled
86352025-04-30T06:43:02.431ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
86362025-04-30T06:43:02.431ZTRACtest_snapshot_basic (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
86372025-04-30T06:43:02.431ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
86382025-04-30T06:43:02.432ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
86392025-04-30T06:43:02.432ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
86402025-04-30T06:43:02.442ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
86412025-04-30T06:43:02.451ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
86422025-04-30T06:43:02.451ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
86432025-04-30T06:43:02.451ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
86442025-04-30T06:43:02.457ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
86452025-04-30T06:43:02.468ZDEBGtest_snapshot_basic (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = [(d1ad44e5-18dd-4193-a70e-bf9042f0e0e1, 0)]
86462025-04-30T06:43:02.468ZTRACtest_snapshot_basic (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
86472025-04-30T06:43:02.468ZTRACtest_snapshot_basic (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: a3 d7 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 ............._mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:53285 req_id = 9af301c4-fcc7-4a93-8af5-45cef137096a
86482025-04-30T06:43:02.469ZDEBGtest_snapshot_basic (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 41943,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:53285 req_id = 9af301c4-fcc7-4a93-8af5-45cef137096a
86492025-04-30T06:43:02.469ZDEBGtest_snapshot_basic (store): query key key = _mgs._tcp
86502025-04-30T06:43:02.469ZDEBGtest_snapshot_basic (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
86512025-04-30T06:43:02.469ZDEBGtest_snapshot_basic (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:53285 query = LowerQuery { name: LowerName(Name("_mgs._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 41502, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = 9af301c4-fcc7-4a93-8af5-45cef137096a
86522025-04-30T06:43:02.470ZDEBGtest_snapshot_basic (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 41502, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 4247, tv_nsec: 268932049 } })
86532025-04-30T06:43:02.470ZTRACtest_snapshot_basic (DnsResolver): lookup_service_targets: looking up SRV target
86542025-04-30T06:43:02.470ZTRACtest_snapshot_basic (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: 99 85 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e ............-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:65261 req_id = 6bc59a21-0766-4136-95eb-a6e69fe2d1db
86552025-04-30T06:43:02.470ZDEBGtest_snapshot_basic (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 39301,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:65261 req_id = 6bc59a21-0766-4136-95eb-a6e69fe2d1db
86562025-04-30T06:43:02.470ZDEBGtest_snapshot_basic (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
86572025-04-30T06:43:02.470ZDEBGtest_snapshot_basic (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:65261 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = 6bc59a21-0766-4136-95eb-a6e69fe2d1db
86582025-04-30T06:43:02.574ZTRACtest_snapshot_basic (DnsResolver): lookup_all_socket_v6 srv dns_name = _clickhouse-admin-keeper._tcp.control-plane.oxide.internal
86592025-04-30T06:43:02.576ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
86602025-04-30T06:43:02.576ZTRACtest_snapshot_basic (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
86612025-04-30T06:43:02.576ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
86622025-04-30T06:43:02.576ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {} }
86632025-04-30T06:43:02.577ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
86642025-04-30T06:43:02.580ZTRACtest_snapshot_basic (dns): buffer buffer = Length: 87 (0x57) bytes\n0000: 77 a9 01 00 00 01 00 00 00 00 00 01 18 5f 63 6c w............_cl\n0010: 69 63 6b 68 6f 75 73 65 2d 61 64 6d 69 6e 2d 6b ickhouse-admin-k\n0020: 65 65 70 65 72 04 5f 74 63 70 0d 63 6f 6e 74 72 eeper._tcp.contr\n0030: 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 69 ol-plane.oxide.i\n0040: 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 04 nternal..!....).\n0050: d0 00 00 00 00 00 00 ....... kind = dns peer_addr = [::1]:53230 req_id = e0b5d048-54ff-497a-8998-3627f291620c
86652025-04-30T06:43:02.580ZDEBGtest_snapshot_basic (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 30633,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 24,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 97,\n 100,\n 109,\n 105,\n 110,\n 45,\n 107,\n 101,\n 101,\n 112,\n 101,\n 114,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:53230 req_id = e0b5d048-54ff-497a-8998-3627f291620c
86662025-04-30T06:43:02.580ZDEBGtest_snapshot_basic (store): query key key = _clickhouse-admin-keeper._tcp
86672025-04-30T06:43:02.581ZERROtest_snapshot_basic (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 30633, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } NXDOMAIN: no records found for name: "_clickhouse-admin-keeper._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:53230 req_id = e0b5d048-54ff-497a-8998-3627f291620c
86682025-04-30T06:43:02.582ZDEBGtest_snapshot_basic (BackgroundTasks): begin collection background_task = inventory_collection
86692025-04-30T06:43:02.582ZDEBGtest_snapshot_basic (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:41502
86702025-04-30T06:43:02.582ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/ignition
86712025-04-30T06:43:02.584ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
86722025-04-30T06:43:02.595ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
86732025-04-30T06:43:02.595ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
86742025-04-30T06:43:02.595ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {} }
86752025-04-30T06:43:02.595ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
86762025-04-30T06:43:02.614ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
86772025-04-30T06:43:02.615ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {} }
86782025-04-30T06:43:02.616ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
86792025-04-30T06:43:02.668ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "c4a23ab4-0dd2-4154-b21c-37df2e359418", "content-length": "835", "date": "Wed, 30 Apr 2025 06:43:02 GMT"} })
86802025-04-30T06:43:02.668ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/0
86812025-04-30T06:43:02.777ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "f153198b-9392-41fb-8aa9-6dc900961130", "content-length": "734", "date": "Wed, 30 Apr 2025 06:43:02 GMT"} })
86822025-04-30T06:43:02.778ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/0/component/sp/caboose?firmware_slot=0
86832025-04-30T06:43:02.789ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "e5ad3e0a-b209-4929-8d90-1ad3d7ccd0a1", "content-length": "111", "date": "Wed, 30 Apr 2025 06:43:02 GMT"} })
86842025-04-30T06:43:02.789ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/0/component/sp/caboose?firmware_slot=1
86852025-04-30T06:43:02.795ZDEBGtest_snapshot_basic (collection-task): sent timer-based collection request to the collection task collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
86862025-04-30T06:43:02.795ZDEBGtest_snapshot_basic (collection-task): collecting from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
86872025-04-30T06:43:02.798ZTRACtest_snapshot_basic (collection-task): sent collection request to producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
86882025-04-30T06:43:02.800ZDEBGtest_snapshot_basic (collection-task): collected results from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 n_results = 1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
86892025-04-30T06:43:02.800ZTRACtest_snapshot_basic (collection-task): forwarded results to main collection loop collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
86902025-04-30T06:43:02.800ZTRACtest_snapshot_basic (collection-task): top of inner collection loop, waiting for next request collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
86912025-04-30T06:43:02.801ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "baf7059b-4707-46cf-bb62-2e80bd62218d", "content-length": "111", "date": "Wed, 30 Apr 2025 06:43:02 GMT"} })
86922025-04-30T06:43:02.801ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/0/component/rot/caboose?firmware_slot=0
86932025-04-30T06:43:02.825ZDEBGtest_snapshot_basic: attempt to fetch latest inventory collection result = Ok("found")
86942025-04-30T06:43:02.829ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "51025b85-d587-48c2-aaa9-18b5573ef571", "content-length": "170", "date": "Wed, 30 Apr 2025 06:43:02 GMT"} })
86952025-04-30T06:43:02.829ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/0/component/rot/caboose?firmware_slot=1
86962025-04-30T06:43:02.834ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "0280a62d-b208-4a0a-b60c-1005a9693e07", "content-length": "170", "date": "Wed, 30 Apr 2025 06:43:02 GMT"} })
86972025-04-30T06:43:02.834ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/0/component/stage0/caboose?firmware_slot=0
86982025-04-30T06:43:02.837ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "ada5ec9b-5bf1-4b39-ab53-714bcd4fc3f4", "content-length": "179", "date": "Wed, 30 Apr 2025 06:43:02 GMT"} })
86992025-04-30T06:43:02.837ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/0/component/stage0/caboose?firmware_slot=1
87002025-04-30T06:43:02.841ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "e7b29b30-4afa-459f-81ff-a6de4fa5aae7", "content-length": "179", "date": "Wed, 30 Apr 2025 06:43:02 GMT"} })
87012025-04-30T06:43:02.841ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/0/component/rot/cmpa
87022025-04-30T06:43:02.842ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "af71eb6f-c94c-4c8b-a7cc-a4f375720c9f", "content-length": "702", "date": "Wed, 30 Apr 2025 06:43:02 GMT"} })
87032025-04-30T06:43:02.842ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/0/component/rot/cfpa
87042025-04-30T06:43:02.845ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "fe612f98-7e7c-4cb3-9e2f-dbd39c1d1283", "content-length": "727", "date": "Wed, 30 Apr 2025 06:43:02 GMT"} })
87052025-04-30T06:43:02.845ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/0/component/rot/cfpa
87062025-04-30T06:43:02.846ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "b7baae21-4b3b-4605-a04f-0b49cc38886a", "content-length": "729", "date": "Wed, 30 Apr 2025 06:43:02 GMT"} })
87072025-04-30T06:43:02.846ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/0/component/rot/cfpa
87082025-04-30T06:43:02.847ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "c9c1decc-bae6-40fe-bb1d-0c0a7965f11e", "content-length": "728", "date": "Wed, 30 Apr 2025 06:43:02 GMT"} })
87092025-04-30T06:43:02.848ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/1
87102025-04-30T06:43:02.868ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "178c6f22-2920-415a-a703-59c23e50ebfa", "content-length": "734", "date": "Wed, 30 Apr 2025 06:43:02 GMT"} })
87112025-04-30T06:43:02.869ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/1/component/sp/caboose?firmware_slot=0
87122025-04-30T06:43:02.872ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "e57db99e-4df8-4c09-b779-1b14d8448345", "content-length": "111", "date": "Wed, 30 Apr 2025 06:43:02 GMT"} })
87132025-04-30T06:43:02.872ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/1/component/sp/caboose?firmware_slot=1
87142025-04-30T06:43:02.874ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "156fda7a-c2db-4372-9e49-a88aaa5f4141", "content-length": "111", "date": "Wed, 30 Apr 2025 06:43:02 GMT"} })
87152025-04-30T06:43:02.874ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/1/component/rot/caboose?firmware_slot=0
87162025-04-30T06:43:02.875ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
87172025-04-30T06:43:02.875ZTRACtest_snapshot_basic (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
87182025-04-30T06:43:02.875ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
87192025-04-30T06:43:02.875ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {} }
87202025-04-30T06:43:02.883ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
87212025-04-30T06:43:02.883ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "1f46b748-3d54-4b08-9968-a15049cff4ca", "content-length": "170", "date": "Wed, 30 Apr 2025 06:43:02 GMT"} })
87222025-04-30T06:43:02.883ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/1/component/rot/caboose?firmware_slot=1
87232025-04-30T06:43:02.883ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
87242025-04-30T06:43:02.888ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
87252025-04-30T06:43:02.888ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
87262025-04-30T06:43:02.888ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {} }
87272025-04-30T06:43:02.888ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
87282025-04-30T06:43:02.898ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
87292025-04-30T06:43:02.898ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {} }
87302025-04-30T06:43:02.899ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
87312025-04-30T06:43:02.901ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "7b9f90ed-4e09-4faa-b125-318a9a9a141d", "content-length": "170", "date": "Wed, 30 Apr 2025 06:43:02 GMT"} })
87322025-04-30T06:43:02.901ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/1/component/stage0/caboose?firmware_slot=0
87332025-04-30T06:43:02.928ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "d2f89af6-b37f-49bb-bfad-fa558af04dd4", "content-length": "179", "date": "Wed, 30 Apr 2025 06:43:02 GMT"} })
87342025-04-30T06:43:02.929ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/1/component/stage0/caboose?firmware_slot=1
87352025-04-30T06:43:02.933ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "987fa468-3355-45d9-b938-207f7d4bab4e", "content-length": "179", "date": "Wed, 30 Apr 2025 06:43:02 GMT"} })
87362025-04-30T06:43:02.933ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/1/component/rot/cmpa
87372025-04-30T06:43:02.935ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "35024258-2d6f-4c95-95de-657a66bf98d2", "content-length": "702", "date": "Wed, 30 Apr 2025 06:43:02 GMT"} })
87382025-04-30T06:43:02.935ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/1/component/rot/cfpa
87392025-04-30T06:43:02.937ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "ef87853e-603d-4707-8ba9-0bd81cedb5cf", "content-length": "727", "date": "Wed, 30 Apr 2025 06:43:02 GMT"} })
87402025-04-30T06:43:02.937ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/1/component/rot/cfpa
87412025-04-30T06:43:02.938ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "b80a613c-c933-4562-a982-c991ec190eb0", "content-length": "729", "date": "Wed, 30 Apr 2025 06:43:02 GMT"} })
87422025-04-30T06:43:02.938ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/1/component/rot/cfpa
87432025-04-30T06:43:02.940ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "3af675e7-7d80-4e27-8dcf-ca5d08267526", "content-length": "728", "date": "Wed, 30 Apr 2025 06:43:02 GMT"} })
87442025-04-30T06:43:02.940ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/0
87452025-04-30T06:43:02.941ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "d2456adc-7e2b-4697-8c9e-19b0391c75bc", "content-length": "723", "date": "Wed, 30 Apr 2025 06:43:02 GMT"} })
87462025-04-30T06:43:02.942ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/0/component/sp/caboose?firmware_slot=0
87472025-04-30T06:43:02.944ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "a12d233a-de72-4859-8a33-f9cd0ea05308", "content-length": "109", "date": "Wed, 30 Apr 2025 06:43:02 GMT"} })
87482025-04-30T06:43:02.944ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/0/component/sp/caboose?firmware_slot=1
87492025-04-30T06:43:02.960ZDEBGtest_snapshot_basic: attempt to fetch latest inventory collection result = Ok("found")
87502025-04-30T06:43:02.968ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "c55750be-1da6-42d5-946d-d012ed451f4c", "content-length": "109", "date": "Wed, 30 Apr 2025 06:43:02 GMT"} })
87512025-04-30T06:43:02.968ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/0/component/rot/caboose?firmware_slot=0
87522025-04-30T06:43:02.971ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "4afe7ad7-dcec-4500-b881-2ec02c7fe3a7", "content-length": "169", "date": "Wed, 30 Apr 2025 06:43:02 GMT"} })
87532025-04-30T06:43:02.971ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/0/component/rot/caboose?firmware_slot=1
87542025-04-30T06:43:02.973ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "1ca3b27c-dbb0-4b4a-9db5-8985c0c77ac1", "content-length": "169", "date": "Wed, 30 Apr 2025 06:43:02 GMT"} })
87552025-04-30T06:43:02.974ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/0/component/stage0/caboose?firmware_slot=0
87562025-04-30T06:43:02.977ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "258aa8af-e48c-44f2-9794-78e5d04721ac", "content-length": "178", "date": "Wed, 30 Apr 2025 06:43:02 GMT"} })
87572025-04-30T06:43:02.977ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/0/component/stage0/caboose?firmware_slot=1
87582025-04-30T06:43:02.987ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "02b9056d-b266-4c5e-8d97-d937dca8219d", "content-length": "178", "date": "Wed, 30 Apr 2025 06:43:02 GMT"} })
87592025-04-30T06:43:02.987ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/0/component/rot/cmpa
87602025-04-30T06:43:02.991ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "36211a18-e5b7-4e68-ba75-0ae72db600d2", "content-length": "702", "date": "Wed, 30 Apr 2025 06:43:02 GMT"} })
87612025-04-30T06:43:02.991ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/0/component/rot/cfpa
87622025-04-30T06:43:02.992ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "b25e0bd9-86cd-4dd8-9b29-fab9286115e4", "content-length": "727", "date": "Wed, 30 Apr 2025 06:43:02 GMT"} })
87632025-04-30T06:43:02.992ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/0/component/rot/cfpa
87642025-04-30T06:43:02.994ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "dbdff0a7-19cd-459a-90d9-51da3f76c8aa", "content-length": "729", "date": "Wed, 30 Apr 2025 06:43:02 GMT"} })
87652025-04-30T06:43:02.994ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/0/component/rot/cfpa
87662025-04-30T06:43:02.995ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "75cd0d9c-a0e8-4b47-a9b1-11f76bbf0324", "content-length": "728", "date": "Wed, 30 Apr 2025 06:43:02 GMT"} })
87672025-04-30T06:43:02.995ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/1
87682025-04-30T06:43:02.996ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "c6cc2e0a-fc91-435b-88e6-b5e169e1ad2b", "content-length": "723", "date": "Wed, 30 Apr 2025 06:43:02 GMT"} })
87692025-04-30T06:43:03.004ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/1/component/sp/caboose?firmware_slot=0
87702025-04-30T06:43:03.008ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "c455647c-0283-4341-9e2b-fecc7fc5fb09", "content-length": "109", "date": "Wed, 30 Apr 2025 06:43:02 GMT"} })
87712025-04-30T06:43:03.008ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/1/component/sp/caboose?firmware_slot=1
87722025-04-30T06:43:03.010ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "1046ab82-47f3-42f4-be95-3a92c038b241", "content-length": "109", "date": "Wed, 30 Apr 2025 06:43:02 GMT"} })
87732025-04-30T06:43:03.011ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/1/component/rot/caboose?firmware_slot=0
87742025-04-30T06:43:03.013ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "ea84239e-ca6d-4a1b-b569-0be9f10aebcc", "content-length": "169", "date": "Wed, 30 Apr 2025 06:43:02 GMT"} })
87752025-04-30T06:43:03.013ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/1/component/rot/caboose?firmware_slot=1
87762025-04-30T06:43:03.023ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
87772025-04-30T06:43:03.023ZTRACtest_snapshot_basic (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
87782025-04-30T06:43:03.023ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
87792025-04-30T06:43:03.023ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {} }
87802025-04-30T06:43:03.024ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
87812025-04-30T06:43:03.025ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "1f9e56f2-c997-45a4-8f15-db7e9428ff4f", "content-length": "169", "date": "Wed, 30 Apr 2025 06:43:02 GMT"} })
87822025-04-30T06:43:03.025ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/1/component/stage0/caboose?firmware_slot=0
87832025-04-30T06:43:03.027ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "9166fad1-c06a-480a-9cd3-1e5c191fb668", "content-length": "178", "date": "Wed, 30 Apr 2025 06:43:02 GMT"} })
87842025-04-30T06:43:03.027ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/1/component/stage0/caboose?firmware_slot=1
87852025-04-30T06:43:03.031ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "503fca1e-7bd8-4375-b25e-39aa227918bb", "content-length": "178", "date": "Wed, 30 Apr 2025 06:43:02 GMT"} })
87862025-04-30T06:43:03.032ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/1/component/rot/cmpa
87872025-04-30T06:43:03.032ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
87882025-04-30T06:43:03.038ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
87892025-04-30T06:43:03.038ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
87902025-04-30T06:43:03.038ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {} }
87912025-04-30T06:43:03.039ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
87922025-04-30T06:43:03.039ZDEBGtest_snapshot_basic (results-sink): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
87932025-04-30T06:43:03.039ZDEBGtest_snapshot_basic (results-sink-cluster): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
87942025-04-30T06:43:03.040ZDEBGtest_snapshot_basic (clickhouse-client): unrolling 1 total samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 6ad06c6b-9a6e-4ed1-b3f4-5d2e9da3f8a7
87952025-04-30T06:43:03.041ZTRACtest_snapshot_basic (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 6ad06c6b-9a6e-4ed1-b3f4-5d2e9da3f8a7 n_columns = 4 n_rows = 2 sql = INSERT INTO oximeter.fields_string FORMAT Native
87962025-04-30T06:43:03.042ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "63187bdf-14a6-441c-9d65-80261b9b33c8", "content-length": "702", "date": "Wed, 30 Apr 2025 06:43:02 GMT"} })
87972025-04-30T06:43:03.042ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/1/component/rot/cfpa
87982025-04-30T06:43:03.043ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
87992025-04-30T06:43:03.043ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {} }
88002025-04-30T06:43:03.044ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
88012025-04-30T06:43:03.045ZDEBGtest_snapshot_basic (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 6ad06c6b-9a6e-4ed1-b3f4-5d2e9da3f8a7 n_rows = 2 table_name = fields_string
88022025-04-30T06:43:03.045ZTRACtest_snapshot_basic (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 6ad06c6b-9a6e-4ed1-b3f4-5d2e9da3f8a7 n_columns = 4 n_rows = 1 sql = INSERT INTO oximeter.measurements_i64 FORMAT Native
88032025-04-30T06:43:03.045ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "64dd4176-9807-4afa-9c2f-0d1dd5c59c97", "content-length": "727", "date": "Wed, 30 Apr 2025 06:43:02 GMT"} })
88042025-04-30T06:43:03.045ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/1/component/rot/cfpa
88052025-04-30T06:43:03.047ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "1aca6299-12e0-4fdc-a4a0-796249cc2da3", "content-length": "729", "date": "Wed, 30 Apr 2025 06:43:02 GMT"} })
88062025-04-30T06:43:03.048ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/1/component/rot/cfpa
88072025-04-30T06:43:03.052ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "57990810-b9de-43de-bf88-8c70c09d08ef", "content-length": "728", "date": "Wed, 30 Apr 2025 06:43:02 GMT"} })
88082025-04-30T06:43:03.052ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
88092025-04-30T06:43:03.052ZTRACtest_snapshot_basic (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
88102025-04-30T06:43:03.052ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
88112025-04-30T06:43:03.052ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
88122025-04-30T06:43:03.053ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
88132025-04-30T06:43:03.054ZDEBGtest_snapshot_basic (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 6ad06c6b-9a6e-4ed1-b3f4-5d2e9da3f8a7 n_rows = 1 table_name = measurements_i64
88142025-04-30T06:43:03.054ZTRACtest_snapshot_basic (results-sink): successfully inserted samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
88152025-04-30T06:43:03.057ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
88162025-04-30T06:43:03.062ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
88172025-04-30T06:43:03.062ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
88182025-04-30T06:43:03.062ZTRACtest_snapshot_basic (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
88192025-04-30T06:43:03.062ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
88202025-04-30T06:43:03.062ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
88212025-04-30T06:43:03.062ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
88222025-04-30T06:43:03.067ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
88232025-04-30T06:43:03.070ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
88242025-04-30T06:43:03.070ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
88252025-04-30T06:43:03.070ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
88262025-04-30T06:43:03.071ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
88272025-04-30T06:43:03.135ZDEBGtest_snapshot_basic (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:41923
88282025-04-30T06:43:03.135ZDEBGtest_snapshot_basic (BackgroundTasks): client request SledAgent = http://[::1]:41923 background_task = inventory_collection body = None method = GET uri = http://[::1]:41923/inventory
88292025-04-30T06:43:03.136ZINFOtest_snapshot_basic (dropshot): accepted connection local_addr = [::1]:41923 remote_addr = [::1]:35006 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
88302025-04-30T06:43:03.136ZTRACtest_snapshot_basic (dropshot): incoming request local_addr = [::1]:41923 method = GET remote_addr = [::1]:35006 req_id = 0a086648-ad35-4167-a29a-7087227e6482 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
88312025-04-30T06:43:03.137ZINFOtest_snapshot_basic (dropshot): request completed latency_us = 908 local_addr = [::1]:41923 method = GET remote_addr = [::1]:35006 req_id = 0a086648-ad35-4167-a29a-7087227e6482 response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
88322025-04-30T06:43:03.138ZDEBGtest_snapshot_basic (BackgroundTasks): client response SledAgent = http://[::1]:41923 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:41923/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "0a086648-ad35-4167-a29a-7087227e6482", "content-length": "3141", "date": "Wed, 30 Apr 2025 06:43:02 GMT"} })
88332025-04-30T06:43:03.140ZDEBGtest_snapshot_basic (BackgroundTasks): begin collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
88342025-04-30T06:43:03.140ZDEBGtest_snapshot_basic (BackgroundTasks): end collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
88352025-04-30T06:43:03.140ZDEBGtest_snapshot_basic (BackgroundTasks): finished collection background_task = inventory_collection
88362025-04-30T06:43:03.144ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
88372025-04-30T06:43:03.144ZTRACtest_snapshot_basic (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
88382025-04-30T06:43:03.144ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
88392025-04-30T06:43:03.144ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
88402025-04-30T06:43:03.144ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
88412025-04-30T06:43:03.144ZWARNtest_snapshot_basic (results-sink-cluster): failed to insert some results into metric DB collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 error = Failed to check out connection to database: No backends found for this service file = oximeter/collector/src/results_sink.rs:92
88422025-04-30T06:43:03.153ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
88432025-04-30T06:43:03.158ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
88442025-04-30T06:43:03.165ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
88452025-04-30T06:43:03.165ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
88462025-04-30T06:43:03.166ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
88472025-04-30T06:43:03.197ZDEBGtest_snapshot_basic: attempt to fetch latest inventory collection result = Ok("found")
88482025-04-30T06:43:03.259ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
88492025-04-30T06:43:03.260ZTRACtest_snapshot_basic (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
88502025-04-30T06:43:03.260ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
88512025-04-30T06:43:03.260ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {} }
88522025-04-30T06:43:03.268ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
88532025-04-30T06:43:03.271ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
88542025-04-30T06:43:03.277ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
88552025-04-30T06:43:03.277ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
88562025-04-30T06:43:03.277ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {} }
88572025-04-30T06:43:03.278ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
88582025-04-30T06:43:03.312ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
88592025-04-30T06:43:03.312ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {} }
88602025-04-30T06:43:03.313ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
88612025-04-30T06:43:03.313ZWARNtest_snapshot_basic (datastore): Non-retryable transaction failure err = DatabaseError(UniqueViolation, "duplicate key value violates unique constraint \\"inv_caboose_pkey\\"") transaction = inventory_insert_collection
88622025-04-30T06:43:03.314ZWARNtest_snapshot_basic (BackgroundTasks): inventory collection failed background_task = inventory_collection error = failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint "inv_caboose_pkey"
88632025-04-30T06:43:03.314ZDEBGtest_snapshot_basic (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 884.097206ms iteration = 29 status = {"error":"failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint \\"inv_caboose_pkey\\""}
88642025-04-30T06:43:03.314ZDEBGtest_snapshot_basic (BackgroundTasks): activating background_task = inventory_collection iteration = 30 reason = Signaled
88652025-04-30T06:43:03.314ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
88662025-04-30T06:43:03.315ZTRACtest_snapshot_basic (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
88672025-04-30T06:43:03.315ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
88682025-04-30T06:43:03.315ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
88692025-04-30T06:43:03.315ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
88702025-04-30T06:43:03.318ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
88712025-04-30T06:43:03.321ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
88722025-04-30T06:43:03.321ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
88732025-04-30T06:43:03.321ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
88742025-04-30T06:43:03.322ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
88752025-04-30T06:43:03.325ZDEBGtest_snapshot_basic (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = [(d1ad44e5-18dd-4193-a70e-bf9042f0e0e1, 0)]
88762025-04-30T06:43:03.325ZTRACtest_snapshot_basic (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
88772025-04-30T06:43:03.334ZTRACtest_snapshot_basic (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: f2 45 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 .E..........._mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:62754 req_id = cfa58a0b-8a22-4397-b83a-8eeebefdc616
88782025-04-30T06:43:03.334ZDEBGtest_snapshot_basic (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 62021,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:62754 req_id = cfa58a0b-8a22-4397-b83a-8eeebefdc616
88792025-04-30T06:43:03.335ZDEBGtest_snapshot_basic (store): query key key = _mgs._tcp
88802025-04-30T06:43:03.335ZDEBGtest_snapshot_basic (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
88812025-04-30T06:43:03.335ZDEBGtest_snapshot_basic (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:62754 query = LowerQuery { name: LowerName(Name("_mgs._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 41502, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = cfa58a0b-8a22-4397-b83a-8eeebefdc616
88822025-04-30T06:43:03.336ZDEBGtest_snapshot_basic (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 41502, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 4248, tv_nsec: 134736497 } })
88832025-04-30T06:43:03.336ZTRACtest_snapshot_basic (DnsResolver): lookup_service_targets: looking up SRV target
88842025-04-30T06:43:03.337ZTRACtest_snapshot_basic (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: ed 21 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e .!..........-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:53580 req_id = 61fe95ac-6b91-4012-93bb-c58b210a0818
88852025-04-30T06:43:03.337ZDEBGtest_snapshot_basic (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 60705,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:53580 req_id = 61fe95ac-6b91-4012-93bb-c58b210a0818
88862025-04-30T06:43:03.337ZDEBGtest_snapshot_basic (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
88872025-04-30T06:43:03.337ZDEBGtest_snapshot_basic (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:53580 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = 61fe95ac-6b91-4012-93bb-c58b210a0818
88882025-04-30T06:43:03.414ZTRACtest_snapshot_basic (DnsResolver): lookup_all_socket_v6 srv dns_name = _clickhouse-admin-keeper._tcp.control-plane.oxide.internal
88892025-04-30T06:43:03.415ZTRACtest_snapshot_basic (dns): buffer buffer = Length: 87 (0x57) bytes\n0000: 5a 51 01 00 00 01 00 00 00 00 00 01 18 5f 63 6c ZQ..........._cl\n0010: 69 63 6b 68 6f 75 73 65 2d 61 64 6d 69 6e 2d 6b ickhouse-admin-k\n0020: 65 65 70 65 72 04 5f 74 63 70 0d 63 6f 6e 74 72 eeper._tcp.contr\n0030: 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 69 ol-plane.oxide.i\n0040: 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 04 nternal..!....).\n0050: d0 00 00 00 00 00 00 ....... kind = dns peer_addr = [::1]:54050 req_id = 6afe3477-40e9-48be-8595-0f2e4f763440
88902025-04-30T06:43:03.415ZDEBGtest_snapshot_basic (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 23121,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 24,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 97,\n 100,\n 109,\n 105,\n 110,\n 45,\n 107,\n 101,\n 101,\n 112,\n 101,\n 114,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:54050 req_id = 6afe3477-40e9-48be-8595-0f2e4f763440
88912025-04-30T06:43:03.415ZDEBGtest_snapshot_basic (store): query key key = _clickhouse-admin-keeper._tcp
88922025-04-30T06:43:03.415ZERROtest_snapshot_basic (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 23121, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } NXDOMAIN: no records found for name: "_clickhouse-admin-keeper._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:54050 req_id = 6afe3477-40e9-48be-8595-0f2e4f763440
88932025-04-30T06:43:03.416ZDEBGtest_snapshot_basic (BackgroundTasks): begin collection background_task = inventory_collection
88942025-04-30T06:43:03.416ZDEBGtest_snapshot_basic (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:41502
88952025-04-30T06:43:03.416ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/ignition
88962025-04-30T06:43:03.421ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "5233916d-c607-4895-8c89-b0d93dd203a3", "content-length": "835", "date": "Wed, 30 Apr 2025 06:43:02 GMT"} })
88972025-04-30T06:43:03.422ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/0
88982025-04-30T06:43:03.427ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "283728e5-7f4b-46a9-93ef-58f5c7a41650", "content-length": "734", "date": "Wed, 30 Apr 2025 06:43:02 GMT"} })
88992025-04-30T06:43:03.428ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/0/component/sp/caboose?firmware_slot=0
89002025-04-30T06:43:03.444ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "a974e599-ff6e-42de-840c-c51b4c759b84", "content-length": "111", "date": "Wed, 30 Apr 2025 06:43:02 GMT"} })
89012025-04-30T06:43:03.447ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/0/component/sp/caboose?firmware_slot=1
89022025-04-30T06:43:03.452ZDEBGtest_snapshot_basic: attempt to fetch latest inventory collection result = Ok("found")
89032025-04-30T06:43:03.468ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "a31aceea-dee1-413a-9fd3-00818a0e1f97", "content-length": "111", "date": "Wed, 30 Apr 2025 06:43:02 GMT"} })
89042025-04-30T06:43:03.468ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/0/component/rot/caboose?firmware_slot=0
89052025-04-30T06:43:03.471ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "848b21ef-fd39-46fc-9efb-406544424578", "content-length": "170", "date": "Wed, 30 Apr 2025 06:43:02 GMT"} })
89062025-04-30T06:43:03.471ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/0/component/rot/caboose?firmware_slot=1
89072025-04-30T06:43:03.484ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "211fcbb4-d488-42d0-9913-500f988a8a4e", "content-length": "170", "date": "Wed, 30 Apr 2025 06:43:02 GMT"} })
89082025-04-30T06:43:03.484ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/0/component/stage0/caboose?firmware_slot=0
89092025-04-30T06:43:03.488ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "ea63b52b-5c73-4e95-b40d-8d925f8e104c", "content-length": "179", "date": "Wed, 30 Apr 2025 06:43:02 GMT"} })
89102025-04-30T06:43:03.488ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/0/component/stage0/caboose?firmware_slot=1
89112025-04-30T06:43:03.498ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "fbf9ac03-6e17-48d3-97dc-57cf58bd4c10", "content-length": "179", "date": "Wed, 30 Apr 2025 06:43:02 GMT"} })
89122025-04-30T06:43:03.498ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/0/component/rot/cmpa
89132025-04-30T06:43:03.499ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "51f98be9-ea9c-4f9b-8d20-6577af02dbd7", "content-length": "702", "date": "Wed, 30 Apr 2025 06:43:02 GMT"} })
89142025-04-30T06:43:03.499ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/0/component/rot/cfpa
89152025-04-30T06:43:03.500ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "7559398d-40cf-473f-9958-6a3ab9a37d76", "content-length": "727", "date": "Wed, 30 Apr 2025 06:43:02 GMT"} })
89162025-04-30T06:43:03.500ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/0/component/rot/cfpa
89172025-04-30T06:43:03.502ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "ac5e04a7-2775-40d7-bb74-97b017ea108c", "content-length": "729", "date": "Wed, 30 Apr 2025 06:43:02 GMT"} })
89182025-04-30T06:43:03.502ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/0/component/rot/cfpa
89192025-04-30T06:43:03.502ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
89202025-04-30T06:43:03.502ZTRACtest_snapshot_basic (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
89212025-04-30T06:43:03.503ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
89222025-04-30T06:43:03.503ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {} }
89232025-04-30T06:43:03.503ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
89242025-04-30T06:43:03.505ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "65e5ffad-a6d3-4868-aafe-3e46d9647aa3", "content-length": "728", "date": "Wed, 30 Apr 2025 06:43:02 GMT"} })
89252025-04-30T06:43:03.505ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/1
89262025-04-30T06:43:03.506ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
89272025-04-30T06:43:03.511ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
89282025-04-30T06:43:03.511ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
89292025-04-30T06:43:03.511ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {} }
89302025-04-30T06:43:03.512ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
89312025-04-30T06:43:03.517ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
89322025-04-30T06:43:03.517ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {} }
89332025-04-30T06:43:03.518ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
89342025-04-30T06:43:03.528ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "bfd3ffd8-edc4-46f3-84f3-32f6cb682d88", "content-length": "734", "date": "Wed, 30 Apr 2025 06:43:02 GMT"} })
89352025-04-30T06:43:03.529ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/1/component/sp/caboose?firmware_slot=0
89362025-04-30T06:43:03.585ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "d78eaa68-2022-40f0-9d43-af15ff6f1bac", "content-length": "111", "date": "Wed, 30 Apr 2025 06:43:02 GMT"} })
89372025-04-30T06:43:03.585ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/1/component/sp/caboose?firmware_slot=1
89382025-04-30T06:43:03.592ZDEBGtest_snapshot_basic: attempt to fetch latest inventory collection result = Ok("found")
89392025-04-30T06:43:03.594ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "e7518fc6-a5cd-4e19-8133-16f5bae9686a", "content-length": "111", "date": "Wed, 30 Apr 2025 06:43:02 GMT"} })
89402025-04-30T06:43:03.594ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/1/component/rot/caboose?firmware_slot=0
89412025-04-30T06:43:03.599ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "5ebe2cab-fde3-4de6-a3ab-6047349bdf2a", "content-length": "170", "date": "Wed, 30 Apr 2025 06:43:02 GMT"} })
89422025-04-30T06:43:03.600ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/1/component/rot/caboose?firmware_slot=1
89432025-04-30T06:43:03.602ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "62411f5c-9407-4722-93d7-026a77f78504", "content-length": "170", "date": "Wed, 30 Apr 2025 06:43:02 GMT"} })
89442025-04-30T06:43:03.603ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/1/component/stage0/caboose?firmware_slot=0
89452025-04-30T06:43:03.605ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "11c1ed11-f93e-4bc8-89d4-4a81ca8daa2d", "content-length": "179", "date": "Wed, 30 Apr 2025 06:43:02 GMT"} })
89462025-04-30T06:43:03.605ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/1/component/stage0/caboose?firmware_slot=1
89472025-04-30T06:43:03.608ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "e5facc25-9d6a-4274-8b63-cb74bd19373f", "content-length": "179", "date": "Wed, 30 Apr 2025 06:43:02 GMT"} })
89482025-04-30T06:43:03.608ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/1/component/rot/cmpa
89492025-04-30T06:43:03.609ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "dcb798ce-c842-4058-b438-6185f6cd8777", "content-length": "702", "date": "Wed, 30 Apr 2025 06:43:02 GMT"} })
89502025-04-30T06:43:03.609ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/1/component/rot/cfpa
89512025-04-30T06:43:03.610ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "c5f4d0ad-51ed-48f2-949d-2de4ff94ff28", "content-length": "727", "date": "Wed, 30 Apr 2025 06:43:02 GMT"} })
89522025-04-30T06:43:03.610ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/1/component/rot/cfpa
89532025-04-30T06:43:03.611ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "c373b9a9-0519-474f-b87c-15c84a7e4faf", "content-length": "729", "date": "Wed, 30 Apr 2025 06:43:02 GMT"} })
89542025-04-30T06:43:03.611ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/1/component/rot/cfpa
89552025-04-30T06:43:03.613ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "18577d4b-c1c7-462b-a380-5b0bdef1fe69", "content-length": "728", "date": "Wed, 30 Apr 2025 06:43:02 GMT"} })
89562025-04-30T06:43:03.613ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/0
89572025-04-30T06:43:03.615ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "71227edd-e58c-42e2-91d0-18a38fc48194", "content-length": "723", "date": "Wed, 30 Apr 2025 06:43:02 GMT"} })
89582025-04-30T06:43:03.615ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/0/component/sp/caboose?firmware_slot=0
89592025-04-30T06:43:03.617ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "8e097d84-e54a-4c6a-b1ac-bc7b0eb47f3e", "content-length": "109", "date": "Wed, 30 Apr 2025 06:43:02 GMT"} })
89602025-04-30T06:43:03.617ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/0/component/sp/caboose?firmware_slot=1
89612025-04-30T06:43:03.631ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "f46840de-c8bd-4b4e-819b-0277dc50b71b", "content-length": "109", "date": "Wed, 30 Apr 2025 06:43:02 GMT"} })
89622025-04-30T06:43:03.631ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/0/component/rot/caboose?firmware_slot=0
89632025-04-30T06:43:03.633ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "9426655e-a452-4127-a07a-270aa7607d0d", "content-length": "169", "date": "Wed, 30 Apr 2025 06:43:02 GMT"} })
89642025-04-30T06:43:03.633ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/0/component/rot/caboose?firmware_slot=1
89652025-04-30T06:43:03.636ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "81bde565-d4a5-4479-bf81-038b362d8e59", "content-length": "169", "date": "Wed, 30 Apr 2025 06:43:02 GMT"} })
89662025-04-30T06:43:03.636ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/0/component/stage0/caboose?firmware_slot=0
89672025-04-30T06:43:03.639ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "c6afc639-7c5e-4d09-b69f-f5e656d95af1", "content-length": "178", "date": "Wed, 30 Apr 2025 06:43:02 GMT"} })
89682025-04-30T06:43:03.639ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/0/component/stage0/caboose?firmware_slot=1
89692025-04-30T06:43:03.641ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "022178ac-94f7-4656-b0d3-beb95c95adeb", "content-length": "178", "date": "Wed, 30 Apr 2025 06:43:02 GMT"} })
89702025-04-30T06:43:03.641ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/0/component/rot/cmpa
89712025-04-30T06:43:03.642ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "b518d7d0-c7c2-4757-9331-1bd7d403aff9", "content-length": "702", "date": "Wed, 30 Apr 2025 06:43:02 GMT"} })
89722025-04-30T06:43:03.642ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/0/component/rot/cfpa
89732025-04-30T06:43:03.643ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
89742025-04-30T06:43:03.643ZTRACtest_snapshot_basic (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
89752025-04-30T06:43:03.643ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
89762025-04-30T06:43:03.643ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {} }
89772025-04-30T06:43:03.644ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
89782025-04-30T06:43:03.644ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "8cb3d40d-d4b7-467f-b889-662a510175f5", "content-length": "727", "date": "Wed, 30 Apr 2025 06:43:02 GMT"} })
89792025-04-30T06:43:03.644ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/0/component/rot/cfpa
89802025-04-30T06:43:03.645ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "789cab05-e4a5-4343-960e-de1af6b5a878", "content-length": "729", "date": "Wed, 30 Apr 2025 06:43:02 GMT"} })
89812025-04-30T06:43:03.652ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/0/component/rot/cfpa
89822025-04-30T06:43:03.652ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "f1569051-2820-4814-a478-74cf4da65bc8", "content-length": "728", "date": "Wed, 30 Apr 2025 06:43:02 GMT"} })
89832025-04-30T06:43:03.652ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/1
89842025-04-30T06:43:03.652ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
89852025-04-30T06:43:03.676ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
89862025-04-30T06:43:03.676ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
89872025-04-30T06:43:03.676ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {} }
89882025-04-30T06:43:03.677ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
89892025-04-30T06:43:03.680ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "28560eb4-4c7d-437f-ae50-e77f5f98abaa", "content-length": "723", "date": "Wed, 30 Apr 2025 06:43:03 GMT"} })
89902025-04-30T06:43:03.681ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/1/component/sp/caboose?firmware_slot=0
89912025-04-30T06:43:03.681ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
89922025-04-30T06:43:03.681ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {} }
89932025-04-30T06:43:03.681ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
89942025-04-30T06:43:03.684ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "83036389-83e5-4a6c-ab58-37a7ac258fc0", "content-length": "109", "date": "Wed, 30 Apr 2025 06:43:03 GMT"} })
89952025-04-30T06:43:03.684ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/1/component/sp/caboose?firmware_slot=1
89962025-04-30T06:43:03.686ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "d5186786-6de6-4ffb-95a9-2996fb06855b", "content-length": "109", "date": "Wed, 30 Apr 2025 06:43:03 GMT"} })
89972025-04-30T06:43:03.686ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/1/component/rot/caboose?firmware_slot=0
89982025-04-30T06:43:03.688ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "f7d082c0-f1c3-4e16-be15-69e152c278ff", "content-length": "169", "date": "Wed, 30 Apr 2025 06:43:03 GMT"} })
89992025-04-30T06:43:03.688ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/1/component/rot/caboose?firmware_slot=1
90002025-04-30T06:43:03.690ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "b1016030-36c0-4a86-a27c-4f326fc5e619", "content-length": "169", "date": "Wed, 30 Apr 2025 06:43:03 GMT"} })
90012025-04-30T06:43:03.691ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/1/component/stage0/caboose?firmware_slot=0
90022025-04-30T06:43:03.692ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "b3011694-97e3-4748-a9e7-bf95901be806", "content-length": "178", "date": "Wed, 30 Apr 2025 06:43:03 GMT"} })
90032025-04-30T06:43:03.692ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/1/component/stage0/caboose?firmware_slot=1
90042025-04-30T06:43:03.694ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "c6de4a46-ba84-4994-8b0c-88258fe2fb29", "content-length": "178", "date": "Wed, 30 Apr 2025 06:43:03 GMT"} })
90052025-04-30T06:43:03.694ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/1/component/rot/cmpa
90062025-04-30T06:43:03.695ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "0d17a434-fda4-49a7-a8e9-8e34aa57c4a9", "content-length": "702", "date": "Wed, 30 Apr 2025 06:43:03 GMT"} })
90072025-04-30T06:43:03.695ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/1/component/rot/cfpa
90082025-04-30T06:43:03.706ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "3cd9365a-e0fa-4ad1-a9a4-208268141dc9", "content-length": "727", "date": "Wed, 30 Apr 2025 06:43:03 GMT"} })
90092025-04-30T06:43:03.706ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/1/component/rot/cfpa
90102025-04-30T06:43:03.714ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "bf52fd2a-c289-4c83-aa2d-5eb686296c0d", "content-length": "729", "date": "Wed, 30 Apr 2025 06:43:03 GMT"} })
90112025-04-30T06:43:03.714ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/1/component/rot/cfpa
90122025-04-30T06:43:03.716ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "d51af011-4472-46b2-910c-0e9a800028b4", "content-length": "728", "date": "Wed, 30 Apr 2025 06:43:03 GMT"} })
90132025-04-30T06:43:03.716ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
90142025-04-30T06:43:03.716ZTRACtest_snapshot_basic (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
90152025-04-30T06:43:03.716ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
90162025-04-30T06:43:03.716ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
90172025-04-30T06:43:03.717ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
90182025-04-30T06:43:03.721ZINFOtest_snapshot_basic (oximeter-agent): refreshing list of producers from Nexus collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 file = oximeter/collector/src/agent.rs:562
90192025-04-30T06:43:03.722ZDEBGtest_snapshot_basic (oximeter): client request body = None method = GET uri = http://127.0.0.1:61468/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
90202025-04-30T06:43:03.723ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
90212025-04-30T06:43:03.731ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
90222025-04-30T06:43:03.731ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
90232025-04-30T06:43:03.731ZTRACtest_snapshot_basic (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
90242025-04-30T06:43:03.735ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
90252025-04-30T06:43:03.735ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
90262025-04-30T06:43:03.737ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
90272025-04-30T06:43:03.740ZTRACtest_snapshot_basic (dropshot_internal): incoming request local_addr = 127.0.0.1:61468 method = GET remote_addr = 127.0.0.1:64332 req_id = c0f04bbf-6a37-4da2-bf15-fe86f534fdab uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
90282025-04-30T06:43:03.741ZTRACtest_snapshot_basic (dropshot_internal): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:61468 method = GET remote_addr = 127.0.0.1:64332 req_id = c0f04bbf-6a37-4da2-bf15-fe86f534fdab resource = Database uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
90292025-04-30T06:43:03.741ZDEBGtest_snapshot_basic (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:61468 method = GET remote_addr = 127.0.0.1:64332 req_id = c0f04bbf-6a37-4da2-bf15-fe86f534fdab roles = RoleSet { roles: {} } uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
90302025-04-30T06:43:03.744ZDEBGtest_snapshot_basic (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:61468 method = GET remote_addr = 127.0.0.1:64332 req_id = c0f04bbf-6a37-4da2-bf15-fe86f534fdab resource = Database result = Ok(()) uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
90312025-04-30T06:43:03.745ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
90322025-04-30T06:43:03.759ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
90332025-04-30T06:43:03.759ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
90342025-04-30T06:43:03.759ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
90352025-04-30T06:43:03.761ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
90362025-04-30T06:43:03.766ZINFOtest_snapshot_basic (dropshot_internal): request completed latency_us = 23744 local_addr = 127.0.0.1:61468 method = GET remote_addr = 127.0.0.1:64332 req_id = c0f04bbf-6a37-4da2-bf15-fe86f534fdab response_code = 200 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
90372025-04-30T06:43:03.766ZDEBGtest_snapshot_basic (oximeter): client response result = Ok(Response { url: "http://127.0.0.1:61468/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending", status: 200, headers: {"content-type": "application/json", "x-request-id": "c0f04bbf-6a37-4da2-bf15-fe86f534fdab", "content-length": "405", "date": "Wed, 30 Apr 2025 06:43:03 GMT"} })
90382025-04-30T06:43:03.766ZDEBGtest_snapshot_basic (oximeter): client request body = None method = GET uri = http://127.0.0.1:61468/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
90392025-04-30T06:43:03.866ZDEBGtest_snapshot_basic (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:41923
90402025-04-30T06:43:03.868ZDEBGtest_snapshot_basic (BackgroundTasks): client request SledAgent = http://[::1]:41923 background_task = inventory_collection body = None method = GET uri = http://[::1]:41923/inventory
90412025-04-30T06:43:03.869ZINFOtest_snapshot_basic (dropshot): accepted connection local_addr = [::1]:41923 remote_addr = [::1]:47825 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
90422025-04-30T06:43:03.869ZTRACtest_snapshot_basic (dropshot_internal): incoming request local_addr = 127.0.0.1:61468 method = GET remote_addr = 127.0.0.1:64332 req_id = 19128da9-ccb9-4dbe-b8dd-56a6bcfaa941 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
90432025-04-30T06:43:03.869ZDEBGtest_snapshot_basic (collection-task): sent timer-based collection request to the collection task collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
90442025-04-30T06:43:03.870ZTRACtest_snapshot_basic (dropshot_internal): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:61468 method = GET remote_addr = 127.0.0.1:64332 req_id = 19128da9-ccb9-4dbe-b8dd-56a6bcfaa941 resource = Database uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
90452025-04-30T06:43:03.870ZDEBGtest_snapshot_basic (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:61468 method = GET remote_addr = 127.0.0.1:64332 req_id = 19128da9-ccb9-4dbe-b8dd-56a6bcfaa941 roles = RoleSet { roles: {} } uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
90462025-04-30T06:43:03.871ZDEBGtest_snapshot_basic (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:61468 method = GET remote_addr = 127.0.0.1:64332 req_id = 19128da9-ccb9-4dbe-b8dd-56a6bcfaa941 resource = Database result = Ok(()) uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
90472025-04-30T06:43:03.871ZDEBGtest_snapshot_basic (collection-task): collecting from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
90482025-04-30T06:43:03.874ZTRACtest_snapshot_basic (dropshot): incoming request local_addr = [::1]:41923 method = GET remote_addr = [::1]:47825 req_id = ddab6bd4-2e8b-4c0f-bbc7-6100b52f8056 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
90492025-04-30T06:43:03.876ZINFOtest_snapshot_basic (dropshot): request completed latency_us = 1625 local_addr = [::1]:41923 method = GET remote_addr = [::1]:47825 req_id = ddab6bd4-2e8b-4c0f-bbc7-6100b52f8056 response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
90502025-04-30T06:43:03.877ZDEBGtest_snapshot_basic: attempt to fetch latest inventory collection result = Ok("found")
90512025-04-30T06:43:03.879ZTRACtest_snapshot_basic (collection-task): sent collection request to producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
90522025-04-30T06:43:03.880ZDEBGtest_snapshot_basic (collection-task): collected results from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 n_results = 1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
90532025-04-30T06:43:03.880ZTRACtest_snapshot_basic (collection-task): forwarded results to main collection loop collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
90542025-04-30T06:43:03.880ZTRACtest_snapshot_basic (collection-task): top of inner collection loop, waiting for next request collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
90552025-04-30T06:43:03.880ZDEBGtest_snapshot_basic (BackgroundTasks): client response SledAgent = http://[::1]:41923 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:41923/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "ddab6bd4-2e8b-4c0f-bbc7-6100b52f8056", "content-length": "3141", "date": "Wed, 30 Apr 2025 06:43:03 GMT"} })
90562025-04-30T06:43:03.882ZDEBGtest_snapshot_basic (BackgroundTasks): begin collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
90572025-04-30T06:43:03.882ZDEBGtest_snapshot_basic (BackgroundTasks): end collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
90582025-04-30T06:43:03.882ZDEBGtest_snapshot_basic (BackgroundTasks): finished collection background_task = inventory_collection
90592025-04-30T06:43:03.884ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
90602025-04-30T06:43:03.884ZTRACtest_snapshot_basic (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
90612025-04-30T06:43:03.884ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
90622025-04-30T06:43:03.884ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
90632025-04-30T06:43:03.885ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
90642025-04-30T06:43:03.885ZINFOtest_snapshot_basic (dropshot_internal): request completed latency_us = 15957 local_addr = 127.0.0.1:61468 method = GET remote_addr = 127.0.0.1:64332 req_id = 19128da9-ccb9-4dbe-b8dd-56a6bcfaa941 response_code = 200 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
90652025-04-30T06:43:03.894ZDEBGtest_snapshot_basic (oximeter): client response result = Ok(Response { url: "http://127.0.0.1:61468/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D", status: 200, headers: {"content-type": "application/json", "x-request-id": "19128da9-ccb9-4dbe-b8dd-56a6bcfaa941", "content-length": "29", "date": "Wed, 30 Apr 2025 06:43:03 GMT"} })
90662025-04-30T06:43:03.894ZTRACtest_snapshot_basic (oximeter-agent): ignoring request to update existing metric producer, since the endpoint information is the same as the existing collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = 354a52d4-3078-4706-81ce-12c32f6bbeaa
90672025-04-30T06:43:03.894ZTRACtest_snapshot_basic (oximeter-agent): ignoring request to update existing metric producer, since the endpoint information is the same as the existing collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
90682025-04-30T06:43:03.894ZINFOtest_snapshot_basic (oximeter-agent): refreshed list of producers from Nexus collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 file = oximeter/collector/src/agent.rs:637 n_current_tasks = 2 n_pruned_tasks = 0
90692025-04-30T06:43:03.895ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
90702025-04-30T06:43:03.899ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
90712025-04-30T06:43:03.899ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
90722025-04-30T06:43:03.899ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
90732025-04-30T06:43:03.899ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
90742025-04-30T06:43:03.932ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
90752025-04-30T06:43:03.932ZTRACtest_snapshot_basic (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
90762025-04-30T06:43:03.932ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
90772025-04-30T06:43:03.932ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {} }
90782025-04-30T06:43:03.933ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
90792025-04-30T06:43:03.940ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
90802025-04-30T06:43:03.960ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
90812025-04-30T06:43:03.961ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
90822025-04-30T06:43:03.961ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {} }
90832025-04-30T06:43:03.961ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
90842025-04-30T06:43:04.010ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
90852025-04-30T06:43:04.010ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {} }
90862025-04-30T06:43:04.011ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
90872025-04-30T06:43:04.011ZWARNtest_snapshot_basic (datastore): Non-retryable transaction failure err = DatabaseError(UniqueViolation, "duplicate key value violates unique constraint \\"inv_caboose_pkey\\"") transaction = inventory_insert_collection
90882025-04-30T06:43:04.011ZWARNtest_snapshot_basic (BackgroundTasks): inventory collection failed background_task = inventory_collection error = failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint "inv_caboose_pkey"
90892025-04-30T06:43:04.011ZDEBGtest_snapshot_basic (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 696.643647ms iteration = 30 status = {"error":"failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint \\"inv_caboose_pkey\\""}
90902025-04-30T06:43:04.011ZDEBGtest_snapshot_basic (BackgroundTasks): activating background_task = inventory_collection iteration = 31 reason = Signaled
90912025-04-30T06:43:04.011ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
90922025-04-30T06:43:04.011ZTRACtest_snapshot_basic (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
90932025-04-30T06:43:04.011ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
90942025-04-30T06:43:04.011ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
90952025-04-30T06:43:04.012ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
90962025-04-30T06:43:04.016ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
90972025-04-30T06:43:04.020ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
90982025-04-30T06:43:04.020ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
90992025-04-30T06:43:04.020ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
91002025-04-30T06:43:04.020ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
91012025-04-30T06:43:04.027ZDEBGtest_snapshot_basic (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = [(d1ad44e5-18dd-4193-a70e-bf9042f0e0e1, 0)]
91022025-04-30T06:43:04.027ZTRACtest_snapshot_basic (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
91032025-04-30T06:43:04.027ZTRACtest_snapshot_basic (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: a1 12 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 ............._mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:53716 req_id = 6445c3c4-001c-4e61-9a89-15ac43410cb6
91042025-04-30T06:43:04.027ZDEBGtest_snapshot_basic (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 41234,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:53716 req_id = 6445c3c4-001c-4e61-9a89-15ac43410cb6
91052025-04-30T06:43:04.027ZDEBGtest_snapshot_basic (store): query key key = _mgs._tcp
91062025-04-30T06:43:04.028ZDEBGtest_snapshot_basic (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
91072025-04-30T06:43:04.028ZDEBGtest_snapshot_basic (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:53716 query = LowerQuery { name: LowerName(Name("_mgs._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 41502, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = 6445c3c4-001c-4e61-9a89-15ac43410cb6
91082025-04-30T06:43:04.028ZDEBGtest_snapshot_basic (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 41502, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 4248, tv_nsec: 825427624 } })
91092025-04-30T06:43:04.028ZTRACtest_snapshot_basic (DnsResolver): lookup_service_targets: looking up SRV target
91102025-04-30T06:43:04.028ZTRACtest_snapshot_basic (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: 45 ee 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e E...........-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:49424 req_id = d7ca0926-f80a-4184-8cf1-bb57ad82e3b8
91112025-04-30T06:43:04.028ZDEBGtest_snapshot_basic (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 17902,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:49424 req_id = d7ca0926-f80a-4184-8cf1-bb57ad82e3b8
91122025-04-30T06:43:04.028ZDEBGtest_snapshot_basic (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
91132025-04-30T06:43:04.028ZDEBGtest_snapshot_basic (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:49424 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = d7ca0926-f80a-4184-8cf1-bb57ad82e3b8
91142025-04-30T06:43:04.094ZTRACtest_snapshot_basic (DnsResolver): lookup_all_socket_v6 srv dns_name = _clickhouse-admin-keeper._tcp.control-plane.oxide.internal
91152025-04-30T06:43:04.095ZDEBGtest_snapshot_basic (results-sink): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
91162025-04-30T06:43:04.095ZDEBGtest_snapshot_basic (results-sink-cluster): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
91172025-04-30T06:43:04.096ZDEBGtest_snapshot_basic (clickhouse-client): unrolling 1 total samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 6ad06c6b-9a6e-4ed1-b3f4-5d2e9da3f8a7
91182025-04-30T06:43:04.097ZTRACtest_snapshot_basic (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 6ad06c6b-9a6e-4ed1-b3f4-5d2e9da3f8a7 n_columns = 4 n_rows = 2 sql = INSERT INTO oximeter.fields_string FORMAT Native
91192025-04-30T06:43:04.097ZTRACtest_snapshot_basic (dns): buffer buffer = Length: 87 (0x57) bytes\n0000: 2c 97 01 00 00 01 00 00 00 00 00 01 18 5f 63 6c ,............_cl\n0010: 69 63 6b 68 6f 75 73 65 2d 61 64 6d 69 6e 2d 6b ickhouse-admin-k\n0020: 65 65 70 65 72 04 5f 74 63 70 0d 63 6f 6e 74 72 eeper._tcp.contr\n0030: 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 69 ol-plane.oxide.i\n0040: 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 04 nternal..!....).\n0050: d0 00 00 00 00 00 00 ....... kind = dns peer_addr = [::1]:64971 req_id = 469163c5-6344-4e6d-aa94-762354688946
91202025-04-30T06:43:04.098ZDEBGtest_snapshot_basic (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 11415,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 24,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 97,\n 100,\n 109,\n 105,\n 110,\n 45,\n 107,\n 101,\n 101,\n 112,\n 101,\n 114,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:64971 req_id = 469163c5-6344-4e6d-aa94-762354688946
91212025-04-30T06:43:04.098ZDEBGtest_snapshot_basic (store): query key key = _clickhouse-admin-keeper._tcp
91222025-04-30T06:43:04.099ZERROtest_snapshot_basic (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 11415, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } NXDOMAIN: no records found for name: "_clickhouse-admin-keeper._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:64971 req_id = 469163c5-6344-4e6d-aa94-762354688946
91232025-04-30T06:43:04.099ZDEBGtest_snapshot_basic (BackgroundTasks): begin collection background_task = inventory_collection
91242025-04-30T06:43:04.099ZDEBGtest_snapshot_basic (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:41502
91252025-04-30T06:43:04.099ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/ignition
91262025-04-30T06:43:04.101ZDEBGtest_snapshot_basic (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 6ad06c6b-9a6e-4ed1-b3f4-5d2e9da3f8a7 n_rows = 2 table_name = fields_string
91272025-04-30T06:43:04.101ZTRACtest_snapshot_basic (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 6ad06c6b-9a6e-4ed1-b3f4-5d2e9da3f8a7 n_columns = 4 n_rows = 1 sql = INSERT INTO oximeter.measurements_i64 FORMAT Native
91282025-04-30T06:43:04.105ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "6f6bb7ad-1d4a-4e94-9ae9-aa630966b121", "content-length": "835", "date": "Wed, 30 Apr 2025 06:43:03 GMT"} })
91292025-04-30T06:43:04.105ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/0
91302025-04-30T06:43:04.106ZDEBGtest_snapshot_basic (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 6ad06c6b-9a6e-4ed1-b3f4-5d2e9da3f8a7 n_rows = 1 table_name = measurements_i64
91312025-04-30T06:43:04.107ZTRACtest_snapshot_basic (results-sink): successfully inserted samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
91322025-04-30T06:43:04.110ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "0fbd886e-556d-4518-94fd-323d44723ffd", "content-length": "734", "date": "Wed, 30 Apr 2025 06:43:03 GMT"} })
91332025-04-30T06:43:04.111ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/0/component/sp/caboose?firmware_slot=0
91342025-04-30T06:43:04.120ZDEBGtest_snapshot_basic: attempt to fetch latest inventory collection result = Ok("found")
91352025-04-30T06:43:04.121ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "fc7e2463-88d0-4ef9-93eb-5ff92773279d", "content-length": "111", "date": "Wed, 30 Apr 2025 06:43:03 GMT"} })
91362025-04-30T06:43:04.121ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/0/component/sp/caboose?firmware_slot=1
91372025-04-30T06:43:04.125ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "54ac86fc-b7b7-4c20-9f31-f45bbbc06ea1", "content-length": "111", "date": "Wed, 30 Apr 2025 06:43:03 GMT"} })
91382025-04-30T06:43:04.126ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/0/component/rot/caboose?firmware_slot=0
91392025-04-30T06:43:04.128ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "d7ec6fe6-4991-4eb3-b15e-d7345c7979cd", "content-length": "170", "date": "Wed, 30 Apr 2025 06:43:03 GMT"} })
91402025-04-30T06:43:04.128ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/0/component/rot/caboose?firmware_slot=1
91412025-04-30T06:43:04.131ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "0050b66a-288b-4d44-8d36-e007ad28e8dd", "content-length": "170", "date": "Wed, 30 Apr 2025 06:43:03 GMT"} })
91422025-04-30T06:43:04.131ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/0/component/stage0/caboose?firmware_slot=0
91432025-04-30T06:43:04.133ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "c7daee76-a2b8-44c3-bf6b-f21e38e9d59d", "content-length": "179", "date": "Wed, 30 Apr 2025 06:43:03 GMT"} })
91442025-04-30T06:43:04.133ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/0/component/stage0/caboose?firmware_slot=1
91452025-04-30T06:43:04.135ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "db53d3b9-5ede-4566-99c9-ab7409a7948b", "content-length": "179", "date": "Wed, 30 Apr 2025 06:43:03 GMT"} })
91462025-04-30T06:43:04.136ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/0/component/rot/cmpa
91472025-04-30T06:43:04.138ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "5c08f13a-dabb-4333-bc2e-1811148ebe38", "content-length": "702", "date": "Wed, 30 Apr 2025 06:43:03 GMT"} })
91482025-04-30T06:43:04.138ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/0/component/rot/cfpa
91492025-04-30T06:43:04.140ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "ac230b96-87d7-4c6d-b94f-b535c2710c40", "content-length": "727", "date": "Wed, 30 Apr 2025 06:43:03 GMT"} })
91502025-04-30T06:43:04.140ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/0/component/rot/cfpa
91512025-04-30T06:43:04.140ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "c4f6fd87-fc15-4069-848c-69a8513829bf", "content-length": "729", "date": "Wed, 30 Apr 2025 06:43:03 GMT"} })
91522025-04-30T06:43:04.141ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/0/component/rot/cfpa
91532025-04-30T06:43:04.144ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "843e6270-11f9-4392-a024-5f053d256035", "content-length": "728", "date": "Wed, 30 Apr 2025 06:43:03 GMT"} })
91542025-04-30T06:43:04.144ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/1
91552025-04-30T06:43:04.146ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "e228987a-1d28-43b2-aa87-b4e35c868c3c", "content-length": "734", "date": "Wed, 30 Apr 2025 06:43:03 GMT"} })
91562025-04-30T06:43:04.147ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/1/component/sp/caboose?firmware_slot=0
91572025-04-30T06:43:04.149ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "90252416-9fab-4182-b276-97d0d9651c02", "content-length": "111", "date": "Wed, 30 Apr 2025 06:43:03 GMT"} })
91582025-04-30T06:43:04.149ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/1/component/sp/caboose?firmware_slot=1
91592025-04-30T06:43:04.151ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "43e9741a-2112-4248-a434-ef06ee33c236", "content-length": "111", "date": "Wed, 30 Apr 2025 06:43:03 GMT"} })
91602025-04-30T06:43:04.152ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/1/component/rot/caboose?firmware_slot=0
91612025-04-30T06:43:04.154ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "3b4965c9-c6ce-4919-9128-61b713458f7f", "content-length": "170", "date": "Wed, 30 Apr 2025 06:43:03 GMT"} })
91622025-04-30T06:43:04.154ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/1/component/rot/caboose?firmware_slot=1
91632025-04-30T06:43:04.159ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "ce70aeb6-db1e-495a-acb3-16315f196ed1", "content-length": "170", "date": "Wed, 30 Apr 2025 06:43:03 GMT"} })
91642025-04-30T06:43:04.161ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/1/component/stage0/caboose?firmware_slot=0
91652025-04-30T06:43:04.165ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "d17436bc-660b-4b36-b6c7-2e75aa111091", "content-length": "179", "date": "Wed, 30 Apr 2025 06:43:03 GMT"} })
91662025-04-30T06:43:04.165ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/1/component/stage0/caboose?firmware_slot=1
91672025-04-30T06:43:04.168ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "abec5fa9-1ad0-4566-a88b-eb5895b8d59e", "content-length": "179", "date": "Wed, 30 Apr 2025 06:43:03 GMT"} })
91682025-04-30T06:43:04.168ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/1/component/rot/cmpa
91692025-04-30T06:43:04.169ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "65015016-802c-498d-8c80-498524a6556a", "content-length": "702", "date": "Wed, 30 Apr 2025 06:43:03 GMT"} })
91702025-04-30T06:43:04.169ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/1/component/rot/cfpa
91712025-04-30T06:43:04.170ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "35cffd83-6066-44c1-a369-5b420243ca7b", "content-length": "727", "date": "Wed, 30 Apr 2025 06:43:03 GMT"} })
91722025-04-30T06:43:04.171ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/1/component/rot/cfpa
91732025-04-30T06:43:04.171ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
91742025-04-30T06:43:04.172ZTRACtest_snapshot_basic (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
91752025-04-30T06:43:04.172ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
91762025-04-30T06:43:04.172ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {} }
91772025-04-30T06:43:04.173ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
91782025-04-30T06:43:04.173ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "f0dae48a-d846-4edd-80e3-eacd67022d0b", "content-length": "729", "date": "Wed, 30 Apr 2025 06:43:03 GMT"} })
91792025-04-30T06:43:04.173ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/1/component/rot/cfpa
91802025-04-30T06:43:04.176ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "d7fedc98-8790-4792-9181-11e9e9427d50", "content-length": "728", "date": "Wed, 30 Apr 2025 06:43:03 GMT"} })
91812025-04-30T06:43:04.176ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/0
91822025-04-30T06:43:04.176ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
91832025-04-30T06:43:04.188ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
91842025-04-30T06:43:04.188ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
91852025-04-30T06:43:04.188ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {} }
91862025-04-30T06:43:04.188ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
91872025-04-30T06:43:04.190ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "13496331-679e-4b01-a6a5-31f0c60795a1", "content-length": "723", "date": "Wed, 30 Apr 2025 06:43:03 GMT"} })
91882025-04-30T06:43:04.191ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/0/component/sp/caboose?firmware_slot=0
91892025-04-30T06:43:04.191ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
91902025-04-30T06:43:04.192ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {} }
91912025-04-30T06:43:04.193ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
91922025-04-30T06:43:04.195ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "4f2b37c7-0fd9-4aec-b31c-bfe76d1edbb9", "content-length": "109", "date": "Wed, 30 Apr 2025 06:43:03 GMT"} })
91932025-04-30T06:43:04.195ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/0/component/sp/caboose?firmware_slot=1
91942025-04-30T06:43:04.197ZWARNtest_snapshot_basic (results-sink-cluster): failed to insert some results into metric DB collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 error = Failed to check out connection to database: No backends found for this service file = oximeter/collector/src/results_sink.rs:92
91952025-04-30T06:43:04.200ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "bbd27850-19fb-4a89-8893-18794684bf90", "content-length": "109", "date": "Wed, 30 Apr 2025 06:43:03 GMT"} })
91962025-04-30T06:43:04.200ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/0/component/rot/caboose?firmware_slot=0
91972025-04-30T06:43:04.203ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "89ba0b75-2f8a-4166-a6f3-9203434c284b", "content-length": "169", "date": "Wed, 30 Apr 2025 06:43:03 GMT"} })
91982025-04-30T06:43:04.204ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/0/component/rot/caboose?firmware_slot=1
91992025-04-30T06:43:04.206ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "bbc82599-a348-4e14-aeb3-1a899a6443a7", "content-length": "169", "date": "Wed, 30 Apr 2025 06:43:03 GMT"} })
92002025-04-30T06:43:04.206ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/0/component/stage0/caboose?firmware_slot=0
92012025-04-30T06:43:04.209ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "d0008385-87e9-4abc-80ab-1c797700ca97", "content-length": "178", "date": "Wed, 30 Apr 2025 06:43:03 GMT"} })
92022025-04-30T06:43:04.209ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/0/component/stage0/caboose?firmware_slot=1
92032025-04-30T06:43:04.212ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "ca2859b7-6c0e-45a3-b485-b3beaa224ebf", "content-length": "178", "date": "Wed, 30 Apr 2025 06:43:03 GMT"} })
92042025-04-30T06:43:04.212ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/0/component/rot/cmpa
92052025-04-30T06:43:04.212ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "cd370c8b-0da6-4ce0-93f8-b640630fd2f1", "content-length": "702", "date": "Wed, 30 Apr 2025 06:43:03 GMT"} })
92062025-04-30T06:43:04.212ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/0/component/rot/cfpa
92072025-04-30T06:43:04.214ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "73e83d3a-6a34-4ba3-9e71-00acb5c1f1ae", "content-length": "727", "date": "Wed, 30 Apr 2025 06:43:03 GMT"} })
92082025-04-30T06:43:04.215ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/0/component/rot/cfpa
92092025-04-30T06:43:04.216ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "87715949-db01-412e-9486-cb3a5831e551", "content-length": "729", "date": "Wed, 30 Apr 2025 06:43:03 GMT"} })
92102025-04-30T06:43:04.216ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/0/component/rot/cfpa
92112025-04-30T06:43:04.217ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "c0f3fa40-5b38-4720-9787-89954207cb20", "content-length": "728", "date": "Wed, 30 Apr 2025 06:43:03 GMT"} })
92122025-04-30T06:43:04.217ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/1
92132025-04-30T06:43:04.222ZDEBGtest_snapshot_basic: attempt to fetch latest inventory collection result = Ok("found")
92142025-04-30T06:43:04.223ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "6144db81-8e53-4949-b5d1-e659e8ba2850", "content-length": "723", "date": "Wed, 30 Apr 2025 06:43:03 GMT"} })
92152025-04-30T06:43:04.223ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/1/component/sp/caboose?firmware_slot=0
92162025-04-30T06:43:04.226ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "e691e5a9-a05e-4f8f-b912-bdab6f900a55", "content-length": "109", "date": "Wed, 30 Apr 2025 06:43:03 GMT"} })
92172025-04-30T06:43:04.226ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/1/component/sp/caboose?firmware_slot=1
92182025-04-30T06:43:04.227ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "6d4e9115-fb2b-4ff8-b5cc-3ccfa6b05f28", "content-length": "109", "date": "Wed, 30 Apr 2025 06:43:03 GMT"} })
92192025-04-30T06:43:04.228ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/1/component/rot/caboose?firmware_slot=0
92202025-04-30T06:43:04.231ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "17996b50-5cc2-4d0d-9eba-876a1a4d2da1", "content-length": "169", "date": "Wed, 30 Apr 2025 06:43:03 GMT"} })
92212025-04-30T06:43:04.232ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/1/component/rot/caboose?firmware_slot=1
92222025-04-30T06:43:04.234ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "42c709a6-e46e-435c-b093-4d6807df790c", "content-length": "169", "date": "Wed, 30 Apr 2025 06:43:03 GMT"} })
92232025-04-30T06:43:04.234ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/1/component/stage0/caboose?firmware_slot=0
92242025-04-30T06:43:04.236ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "fada68a0-d5c3-45a7-8a15-d6cb67aa18ac", "content-length": "178", "date": "Wed, 30 Apr 2025 06:43:03 GMT"} })
92252025-04-30T06:43:04.236ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/1/component/stage0/caboose?firmware_slot=1
92262025-04-30T06:43:04.238ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "71c38359-c8c5-42ea-b971-96e50e9b255e", "content-length": "178", "date": "Wed, 30 Apr 2025 06:43:03 GMT"} })
92272025-04-30T06:43:04.238ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/1/component/rot/cmpa
92282025-04-30T06:43:04.239ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "59cf80e8-17fc-454a-9623-3e6a7fa19c72", "content-length": "702", "date": "Wed, 30 Apr 2025 06:43:03 GMT"} })
92292025-04-30T06:43:04.240ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/1/component/rot/cfpa
92302025-04-30T06:43:04.241ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "502097af-7dd9-42da-b6f5-b3b3189e05c5", "content-length": "727", "date": "Wed, 30 Apr 2025 06:43:03 GMT"} })
92312025-04-30T06:43:04.241ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/1/component/rot/cfpa
92322025-04-30T06:43:04.242ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "7dbc145c-0531-4518-97f5-5f9efbaf8427", "content-length": "729", "date": "Wed, 30 Apr 2025 06:43:03 GMT"} })
92332025-04-30T06:43:04.242ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/1/component/rot/cfpa
92342025-04-30T06:43:04.243ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "99d6565d-2650-4aff-b4d9-16c17738c035", "content-length": "728", "date": "Wed, 30 Apr 2025 06:43:03 GMT"} })
92352025-04-30T06:43:04.243ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
92362025-04-30T06:43:04.243ZTRACtest_snapshot_basic (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
92372025-04-30T06:43:04.243ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
92382025-04-30T06:43:04.243ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
92392025-04-30T06:43:04.244ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
92402025-04-30T06:43:04.246ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
92412025-04-30T06:43:04.252ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
92422025-04-30T06:43:04.252ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
92432025-04-30T06:43:04.252ZTRACtest_snapshot_basic (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
92442025-04-30T06:43:04.252ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
92452025-04-30T06:43:04.252ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
92462025-04-30T06:43:04.253ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
92472025-04-30T06:43:04.256ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
92482025-04-30T06:43:04.262ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
92492025-04-30T06:43:04.262ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
92502025-04-30T06:43:04.262ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
92512025-04-30T06:43:04.262ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
92522025-04-30T06:43:04.339ZDEBGtest_snapshot_basic (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:41923
92532025-04-30T06:43:04.339ZDEBGtest_snapshot_basic (BackgroundTasks): client request SledAgent = http://[::1]:41923 background_task = inventory_collection body = None method = GET uri = http://[::1]:41923/inventory
92542025-04-30T06:43:04.339ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
92552025-04-30T06:43:04.339ZTRACtest_snapshot_basic (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
92562025-04-30T06:43:04.339ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
92572025-04-30T06:43:04.339ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {} }
92582025-04-30T06:43:04.340ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
92592025-04-30T06:43:04.340ZINFOtest_snapshot_basic (dropshot): accepted connection local_addr = [::1]:41923 remote_addr = [::1]:64957 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
92602025-04-30T06:43:04.341ZTRACtest_snapshot_basic (dropshot): incoming request local_addr = [::1]:41923 method = GET remote_addr = [::1]:64957 req_id = f1830c39-f0b7-4d64-a847-55fdc339decf sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
92612025-04-30T06:43:04.342ZINFOtest_snapshot_basic (dropshot): request completed latency_us = 483 local_addr = [::1]:41923 method = GET remote_addr = [::1]:64957 req_id = f1830c39-f0b7-4d64-a847-55fdc339decf response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
92622025-04-30T06:43:04.342ZDEBGtest_snapshot_basic (BackgroundTasks): client response SledAgent = http://[::1]:41923 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:41923/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "f1830c39-f0b7-4d64-a847-55fdc339decf", "content-length": "3141", "date": "Wed, 30 Apr 2025 06:43:03 GMT"} })
92632025-04-30T06:43:04.344ZDEBGtest_snapshot_basic (BackgroundTasks): begin collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
92642025-04-30T06:43:04.344ZDEBGtest_snapshot_basic (BackgroundTasks): end collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
92652025-04-30T06:43:04.344ZDEBGtest_snapshot_basic (BackgroundTasks): finished collection background_task = inventory_collection
92662025-04-30T06:43:04.346ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
92672025-04-30T06:43:04.346ZTRACtest_snapshot_basic (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
92682025-04-30T06:43:04.346ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
92692025-04-30T06:43:04.346ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
92702025-04-30T06:43:04.346ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
92712025-04-30T06:43:04.348ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
92722025-04-30T06:43:04.354ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
92732025-04-30T06:43:04.354ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
92742025-04-30T06:43:04.354ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {} }
92752025-04-30T06:43:04.355ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
92762025-04-30T06:43:04.359ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
92772025-04-30T06:43:04.363ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
92782025-04-30T06:43:04.363ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
92792025-04-30T06:43:04.363ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
92802025-04-30T06:43:04.364ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
92812025-04-30T06:43:04.366ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
92822025-04-30T06:43:04.366ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {} }
92832025-04-30T06:43:04.366ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
92842025-04-30T06:43:04.417ZWARNtest_snapshot_basic (datastore): Non-retryable transaction failure err = DatabaseError(UniqueViolation, "duplicate key value violates unique constraint \\"inv_caboose_pkey\\"") transaction = inventory_insert_collection
92852025-04-30T06:43:04.417ZWARNtest_snapshot_basic (BackgroundTasks): inventory collection failed background_task = inventory_collection error = failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint "inv_caboose_pkey"
92862025-04-30T06:43:04.418ZDEBGtest_snapshot_basic (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 406.098896ms iteration = 31 status = {"error":"failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint \\"inv_caboose_pkey\\""}
92872025-04-30T06:43:04.418ZDEBGtest_snapshot_basic (BackgroundTasks): activating background_task = inventory_collection iteration = 32 reason = Signaled
92882025-04-30T06:43:04.418ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
92892025-04-30T06:43:04.418ZTRACtest_snapshot_basic (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
92902025-04-30T06:43:04.418ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
92912025-04-30T06:43:04.418ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
92922025-04-30T06:43:04.420ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
92932025-04-30T06:43:04.425ZDEBGtest_snapshot_basic: attempt to fetch latest inventory collection result = Ok("found")
92942025-04-30T06:43:04.425ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
92952025-04-30T06:43:04.431ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
92962025-04-30T06:43:04.431ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
92972025-04-30T06:43:04.431ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
92982025-04-30T06:43:04.431ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
92992025-04-30T06:43:04.456ZDEBGtest_snapshot_basic (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = [(d1ad44e5-18dd-4193-a70e-bf9042f0e0e1, 0)]
93002025-04-30T06:43:04.456ZTRACtest_snapshot_basic (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
93012025-04-30T06:43:04.458ZTRACtest_snapshot_basic (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: 2a f6 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 *............_mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:65444 req_id = 662f72dd-1e21-4184-ae87-2bba42b7b184
93022025-04-30T06:43:04.458ZDEBGtest_snapshot_basic (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 10998,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:65444 req_id = 662f72dd-1e21-4184-ae87-2bba42b7b184
93032025-04-30T06:43:04.458ZDEBGtest_snapshot_basic (store): query key key = _mgs._tcp
93042025-04-30T06:43:04.459ZDEBGtest_snapshot_basic (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
93052025-04-30T06:43:04.459ZDEBGtest_snapshot_basic (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:65444 query = LowerQuery { name: LowerName(Name("_mgs._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 41502, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = 662f72dd-1e21-4184-ae87-2bba42b7b184
93062025-04-30T06:43:04.459ZDEBGtest_snapshot_basic (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 41502, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 4249, tv_nsec: 258124544 } })
93072025-04-30T06:43:04.459ZTRACtest_snapshot_basic (DnsResolver): lookup_service_targets: looking up SRV target
93082025-04-30T06:43:04.461ZTRACtest_snapshot_basic (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: 6f b7 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e o...........-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:54523 req_id = 7a7841e7-908e-44ac-916f-005b061b10f8
93092025-04-30T06:43:04.461ZDEBGtest_snapshot_basic (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 28599,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:54523 req_id = 7a7841e7-908e-44ac-916f-005b061b10f8
93102025-04-30T06:43:04.461ZDEBGtest_snapshot_basic (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
93112025-04-30T06:43:04.462ZDEBGtest_snapshot_basic (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:54523 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = 7a7841e7-908e-44ac-916f-005b061b10f8
93122025-04-30T06:43:04.521ZTRACtest_snapshot_basic (DnsResolver): lookup_all_socket_v6 srv dns_name = _clickhouse-admin-keeper._tcp.control-plane.oxide.internal
93132025-04-30T06:43:04.521ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
93142025-04-30T06:43:04.521ZTRACtest_snapshot_basic (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
93152025-04-30T06:43:04.522ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
93162025-04-30T06:43:04.522ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {} }
93172025-04-30T06:43:04.523ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
93182025-04-30T06:43:04.524ZTRACtest_snapshot_basic (dns): buffer buffer = Length: 87 (0x57) bytes\n0000: 8c f6 01 00 00 01 00 00 00 00 00 01 18 5f 63 6c ............._cl\n0010: 69 63 6b 68 6f 75 73 65 2d 61 64 6d 69 6e 2d 6b ickhouse-admin-k\n0020: 65 65 70 65 72 04 5f 74 63 70 0d 63 6f 6e 74 72 eeper._tcp.contr\n0030: 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 69 ol-plane.oxide.i\n0040: 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 04 nternal..!....).\n0050: d0 00 00 00 00 00 00 ....... kind = dns peer_addr = [::1]:62794 req_id = d260bca5-a36e-417c-8cbd-6f33320236ed
93192025-04-30T06:43:04.524ZDEBGtest_snapshot_basic (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 36086,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 24,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 97,\n 100,\n 109,\n 105,\n 110,\n 45,\n 107,\n 101,\n 101,\n 112,\n 101,\n 114,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:62794 req_id = d260bca5-a36e-417c-8cbd-6f33320236ed
93202025-04-30T06:43:04.525ZDEBGtest_snapshot_basic (store): query key key = _clickhouse-admin-keeper._tcp
93212025-04-30T06:43:04.525ZERROtest_snapshot_basic (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 36086, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } NXDOMAIN: no records found for name: "_clickhouse-admin-keeper._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:62794 req_id = d260bca5-a36e-417c-8cbd-6f33320236ed
93222025-04-30T06:43:04.525ZDEBGtest_snapshot_basic (BackgroundTasks): begin collection background_task = inventory_collection
93232025-04-30T06:43:04.526ZDEBGtest_snapshot_basic (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:41502
93242025-04-30T06:43:04.526ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/ignition
93252025-04-30T06:43:04.527ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
93262025-04-30T06:43:04.534ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
93272025-04-30T06:43:04.535ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
93282025-04-30T06:43:04.535ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {} }
93292025-04-30T06:43:04.535ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
93302025-04-30T06:43:04.545ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
93312025-04-30T06:43:04.545ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {} }
93322025-04-30T06:43:04.546ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
93332025-04-30T06:43:04.561ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "e3ac2277-1607-4e54-851a-e6f612a2460f", "content-length": "835", "date": "Wed, 30 Apr 2025 06:43:03 GMT"} })
93342025-04-30T06:43:04.561ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/0
93352025-04-30T06:43:04.576ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "196e3527-2d67-403b-8123-901fe554c6bf", "content-length": "734", "date": "Wed, 30 Apr 2025 06:43:03 GMT"} })
93362025-04-30T06:43:04.576ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/0/component/sp/caboose?firmware_slot=0
93372025-04-30T06:43:04.589ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "e4a4e147-dd35-4ddd-aeaf-0761ce2e7702", "content-length": "111", "date": "Wed, 30 Apr 2025 06:43:03 GMT"} })
93382025-04-30T06:43:04.589ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/0/component/sp/caboose?firmware_slot=1
93392025-04-30T06:43:04.592ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "ef02a3c1-f403-476f-8998-49938066d763", "content-length": "111", "date": "Wed, 30 Apr 2025 06:43:03 GMT"} })
93402025-04-30T06:43:04.592ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/0/component/rot/caboose?firmware_slot=0
93412025-04-30T06:43:04.595ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "cac504f1-265d-43bd-83f9-e38d29bc4098", "content-length": "170", "date": "Wed, 30 Apr 2025 06:43:03 GMT"} })
93422025-04-30T06:43:04.595ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/0/component/rot/caboose?firmware_slot=1
93432025-04-30T06:43:04.599ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "a818597a-eb13-4202-baa2-ab4266252ca9", "content-length": "170", "date": "Wed, 30 Apr 2025 06:43:03 GMT"} })
93442025-04-30T06:43:04.599ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/0/component/stage0/caboose?firmware_slot=0
93452025-04-30T06:43:04.612ZDEBGtest_snapshot_basic: attempt to fetch latest inventory collection result = Ok("found")
93462025-04-30T06:43:04.618ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "ffb5ff22-a533-403a-ad53-1e8704426dd9", "content-length": "179", "date": "Wed, 30 Apr 2025 06:43:03 GMT"} })
93472025-04-30T06:43:04.618ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/0/component/stage0/caboose?firmware_slot=1
93482025-04-30T06:43:04.623ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "f8059122-1ecf-476b-9d05-e2a35e8965a6", "content-length": "179", "date": "Wed, 30 Apr 2025 06:43:03 GMT"} })
93492025-04-30T06:43:04.623ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/0/component/rot/cmpa
93502025-04-30T06:43:04.625ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "9ae97b48-2877-4d74-98fa-236cdee64dca", "content-length": "702", "date": "Wed, 30 Apr 2025 06:43:03 GMT"} })
93512025-04-30T06:43:04.625ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/0/component/rot/cfpa
93522025-04-30T06:43:04.629ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "04984763-afbb-4a75-b51d-c4fab8ef5ef9", "content-length": "727", "date": "Wed, 30 Apr 2025 06:43:03 GMT"} })
93532025-04-30T06:43:04.630ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/0/component/rot/cfpa
93542025-04-30T06:43:04.632ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "cacf0ba1-f141-4753-8bf0-52bfb896ad63", "content-length": "729", "date": "Wed, 30 Apr 2025 06:43:03 GMT"} })
93552025-04-30T06:43:04.633ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/0/component/rot/cfpa
93562025-04-30T06:43:04.634ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "bd853f49-f312-4649-8512-ab26c6db423f", "content-length": "728", "date": "Wed, 30 Apr 2025 06:43:03 GMT"} })
93572025-04-30T06:43:04.634ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/1
93582025-04-30T06:43:04.636ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "b71bbfc6-64c8-4d1b-96a1-c08915b62a0e", "content-length": "734", "date": "Wed, 30 Apr 2025 06:43:03 GMT"} })
93592025-04-30T06:43:04.636ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/1/component/sp/caboose?firmware_slot=0
93602025-04-30T06:43:04.639ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "acf55d31-3e14-43ae-980f-a7448c41e740", "content-length": "111", "date": "Wed, 30 Apr 2025 06:43:03 GMT"} })
93612025-04-30T06:43:04.639ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/1/component/sp/caboose?firmware_slot=1
93622025-04-30T06:43:04.642ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "f59540f8-a4af-4a18-9ee4-573f44aeb5a2", "content-length": "111", "date": "Wed, 30 Apr 2025 06:43:03 GMT"} })
93632025-04-30T06:43:04.642ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/1/component/rot/caboose?firmware_slot=0
93642025-04-30T06:43:04.646ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "85e463b4-1a8a-43b1-bb89-ce12488d7841", "content-length": "170", "date": "Wed, 30 Apr 2025 06:43:03 GMT"} })
93652025-04-30T06:43:04.646ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/1/component/rot/caboose?firmware_slot=1
93662025-04-30T06:43:04.652ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "4e994ead-0874-46e3-92c4-7aee38fa0dcf", "content-length": "170", "date": "Wed, 30 Apr 2025 06:43:03 GMT"} })
93672025-04-30T06:43:04.652ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/1/component/stage0/caboose?firmware_slot=0
93682025-04-30T06:43:04.657ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "e160fe0d-582d-488b-887a-44bc5a8bc160", "content-length": "179", "date": "Wed, 30 Apr 2025 06:43:03 GMT"} })
93692025-04-30T06:43:04.657ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/1/component/stage0/caboose?firmware_slot=1
93702025-04-30T06:43:04.662ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "6cc86af6-4183-42a3-8b78-6c92f08cc6f5", "content-length": "179", "date": "Wed, 30 Apr 2025 06:43:03 GMT"} })
93712025-04-30T06:43:04.662ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/1/component/rot/cmpa
93722025-04-30T06:43:04.663ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
93732025-04-30T06:43:04.663ZTRACtest_snapshot_basic (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
93742025-04-30T06:43:04.663ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
93752025-04-30T06:43:04.663ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {} }
93762025-04-30T06:43:04.665ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
93772025-04-30T06:43:04.667ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "ac4583f7-4e8c-4922-8683-5ca4f3c32608", "content-length": "702", "date": "Wed, 30 Apr 2025 06:43:03 GMT"} })
93782025-04-30T06:43:04.667ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/1/component/rot/cfpa
93792025-04-30T06:43:04.670ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
93802025-04-30T06:43:04.678ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
93812025-04-30T06:43:04.678ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
93822025-04-30T06:43:04.678ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {} }
93832025-04-30T06:43:04.679ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
93842025-04-30T06:43:04.681ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "2b172514-1c24-4e7c-9abd-abb514430500", "content-length": "727", "date": "Wed, 30 Apr 2025 06:43:03 GMT"} })
93852025-04-30T06:43:04.681ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/1/component/rot/cfpa
93862025-04-30T06:43:04.682ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "7168a147-c2a6-4908-9f34-a6e76411e7b0", "content-length": "729", "date": "Wed, 30 Apr 2025 06:43:04 GMT"} })
93872025-04-30T06:43:04.683ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/1/component/rot/cfpa
93882025-04-30T06:43:04.683ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
93892025-04-30T06:43:04.683ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {} }
93902025-04-30T06:43:04.684ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
93912025-04-30T06:43:04.686ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "b624cb00-cba4-4ffe-aa9d-c4b7b02e70b8", "content-length": "728", "date": "Wed, 30 Apr 2025 06:43:04 GMT"} })
93922025-04-30T06:43:04.686ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/0
93932025-04-30T06:43:04.689ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "b49341a8-759b-4c41-9ff0-980a72d31f10", "content-length": "723", "date": "Wed, 30 Apr 2025 06:43:04 GMT"} })
93942025-04-30T06:43:04.689ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/0/component/sp/caboose?firmware_slot=0
93952025-04-30T06:43:04.693ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "0ed98bcd-1750-4702-bada-8abc9dbdba56", "content-length": "109", "date": "Wed, 30 Apr 2025 06:43:04 GMT"} })
93962025-04-30T06:43:04.694ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/0/component/sp/caboose?firmware_slot=1
93972025-04-30T06:43:04.698ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "e25a35fa-d8c9-4e07-b4a7-7d970de142c5", "content-length": "109", "date": "Wed, 30 Apr 2025 06:43:04 GMT"} })
93982025-04-30T06:43:04.698ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/0/component/rot/caboose?firmware_slot=0
93992025-04-30T06:43:04.701ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "3e44cb2c-bfb1-4a39-b9b2-9948854d6aab", "content-length": "169", "date": "Wed, 30 Apr 2025 06:43:04 GMT"} })
94002025-04-30T06:43:04.701ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/0/component/rot/caboose?firmware_slot=1
94012025-04-30T06:43:04.703ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "c6ffae2a-3f67-4f69-9196-ab3933d8ebaa", "content-length": "169", "date": "Wed, 30 Apr 2025 06:43:04 GMT"} })
94022025-04-30T06:43:04.703ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/0/component/stage0/caboose?firmware_slot=0
94032025-04-30T06:43:04.706ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "8ad40dd8-4ef8-4053-a44f-535f0778d241", "content-length": "178", "date": "Wed, 30 Apr 2025 06:43:04 GMT"} })
94042025-04-30T06:43:04.706ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/0/component/stage0/caboose?firmware_slot=1
94052025-04-30T06:43:04.708ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "00a74a38-36d8-4634-b0a9-ce04579acb72", "content-length": "178", "date": "Wed, 30 Apr 2025 06:43:04 GMT"} })
94062025-04-30T06:43:04.708ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/0/component/rot/cmpa
94072025-04-30T06:43:04.709ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "23e5d94f-b008-4f27-9a52-87dd3d8c8959", "content-length": "702", "date": "Wed, 30 Apr 2025 06:43:04 GMT"} })
94082025-04-30T06:43:04.710ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/0/component/rot/cfpa
94092025-04-30T06:43:04.711ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "5fa5d00e-2abe-4e4e-b5e8-9c47055ce8b7", "content-length": "727", "date": "Wed, 30 Apr 2025 06:43:04 GMT"} })
94102025-04-30T06:43:04.711ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/0/component/rot/cfpa
94112025-04-30T06:43:04.712ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "8bda55ce-8d8f-4f02-be11-531a8bbac276", "content-length": "729", "date": "Wed, 30 Apr 2025 06:43:04 GMT"} })
94122025-04-30T06:43:04.712ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/0/component/rot/cfpa
94132025-04-30T06:43:04.715ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "aa2e9a89-d69e-448f-8a58-c85a65af77ca", "content-length": "728", "date": "Wed, 30 Apr 2025 06:43:04 GMT"} })
94142025-04-30T06:43:04.715ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/1
94152025-04-30T06:43:04.716ZDEBGtest_snapshot_basic: attempt to fetch latest inventory collection result = Ok("found")
94162025-04-30T06:43:04.722ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "6e419b8a-440c-4eb4-ad7b-3631130339e1", "content-length": "723", "date": "Wed, 30 Apr 2025 06:43:04 GMT"} })
94172025-04-30T06:43:04.722ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/1/component/sp/caboose?firmware_slot=0
94182025-04-30T06:43:04.725ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "a70cf812-28bf-49b0-8b98-c6408d85d9fa", "content-length": "109", "date": "Wed, 30 Apr 2025 06:43:04 GMT"} })
94192025-04-30T06:43:04.725ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/1/component/sp/caboose?firmware_slot=1
94202025-04-30T06:43:04.729ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "dd16897d-e087-478c-9e14-1ab369fa5596", "content-length": "109", "date": "Wed, 30 Apr 2025 06:43:04 GMT"} })
94212025-04-30T06:43:04.729ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/1/component/rot/caboose?firmware_slot=0
94222025-04-30T06:43:04.733ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "cd711238-4ab2-40b7-a25a-be657fc5f1e8", "content-length": "169", "date": "Wed, 30 Apr 2025 06:43:04 GMT"} })
94232025-04-30T06:43:04.733ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/1/component/rot/caboose?firmware_slot=1
94242025-04-30T06:43:04.735ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "c4fff3d3-7663-453d-9261-5ecc61c49d6c", "content-length": "169", "date": "Wed, 30 Apr 2025 06:43:04 GMT"} })
94252025-04-30T06:43:04.736ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/1/component/stage0/caboose?firmware_slot=0
94262025-04-30T06:43:04.738ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "5799e28b-7d59-49a0-9e66-c253273e389b", "content-length": "178", "date": "Wed, 30 Apr 2025 06:43:04 GMT"} })
94272025-04-30T06:43:04.738ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/1/component/stage0/caboose?firmware_slot=1
94282025-04-30T06:43:04.742ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "0f0fd957-35d1-4542-bc8b-7f1764dfd12b", "content-length": "178", "date": "Wed, 30 Apr 2025 06:43:04 GMT"} })
94292025-04-30T06:43:04.742ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/1/component/rot/cmpa
94302025-04-30T06:43:04.743ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "a85790ca-bf06-43a0-b98b-619d1852712c", "content-length": "702", "date": "Wed, 30 Apr 2025 06:43:04 GMT"} })
94312025-04-30T06:43:04.743ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/1/component/rot/cfpa
94322025-04-30T06:43:04.745ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "c57976e6-c59e-446e-9c1f-1494a918741d", "content-length": "727", "date": "Wed, 30 Apr 2025 06:43:04 GMT"} })
94332025-04-30T06:43:04.745ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/1/component/rot/cfpa
94342025-04-30T06:43:04.747ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "a1e34a32-5610-473f-ae61-4dbedec43066", "content-length": "729", "date": "Wed, 30 Apr 2025 06:43:04 GMT"} })
94352025-04-30T06:43:04.747ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/1/component/rot/cfpa
94362025-04-30T06:43:04.749ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "37e52124-49ab-479e-a3c5-3268164501f4", "content-length": "728", "date": "Wed, 30 Apr 2025 06:43:04 GMT"} })
94372025-04-30T06:43:04.749ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
94382025-04-30T06:43:04.749ZTRACtest_snapshot_basic (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
94392025-04-30T06:43:04.749ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
94402025-04-30T06:43:04.749ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
94412025-04-30T06:43:04.750ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
94422025-04-30T06:43:04.753ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
94432025-04-30T06:43:04.765ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
94442025-04-30T06:43:04.765ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
94452025-04-30T06:43:04.765ZTRACtest_snapshot_basic (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
94462025-04-30T06:43:04.765ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
94472025-04-30T06:43:04.765ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
94482025-04-30T06:43:04.765ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
94492025-04-30T06:43:04.767ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
94502025-04-30T06:43:04.767ZTRACtest_snapshot_basic (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
94512025-04-30T06:43:04.767ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
94522025-04-30T06:43:04.767ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {} }
94532025-04-30T06:43:04.768ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
94542025-04-30T06:43:04.770ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
94552025-04-30T06:43:04.778ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
94562025-04-30T06:43:04.779ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
94572025-04-30T06:43:04.779ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
94582025-04-30T06:43:04.780ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
94592025-04-30T06:43:04.782ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
94602025-04-30T06:43:04.788ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
94612025-04-30T06:43:04.788ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
94622025-04-30T06:43:04.789ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {} }
94632025-04-30T06:43:04.789ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
94642025-04-30T06:43:04.791ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
94652025-04-30T06:43:04.791ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {} }
94662025-04-30T06:43:04.792ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
94672025-04-30T06:43:04.860ZDEBGtest_snapshot_basic (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:41923
94682025-04-30T06:43:04.860ZDEBGtest_snapshot_basic (BackgroundTasks): client request SledAgent = http://[::1]:41923 background_task = inventory_collection body = None method = GET uri = http://[::1]:41923/inventory
94692025-04-30T06:43:04.860ZINFOtest_snapshot_basic (dropshot): accepted connection local_addr = [::1]:41923 remote_addr = [::1]:56836 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
94702025-04-30T06:43:04.861ZDEBGtest_snapshot_basic (collection-task): sent timer-based collection request to the collection task collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
94712025-04-30T06:43:04.861ZDEBGtest_snapshot_basic (collection-task): collecting from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
94722025-04-30T06:43:04.861ZTRACtest_snapshot_basic (dropshot): incoming request local_addr = [::1]:41923 method = GET remote_addr = [::1]:56836 req_id = d676520d-1179-471f-b7cc-b6c8f0b21205 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
94732025-04-30T06:43:04.863ZINFOtest_snapshot_basic (dropshot): request completed latency_us = 1705 local_addr = [::1]:41923 method = GET remote_addr = [::1]:56836 req_id = d676520d-1179-471f-b7cc-b6c8f0b21205 response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
94742025-04-30T06:43:04.865ZTRACtest_snapshot_basic (collection-task): sent collection request to producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
94752025-04-30T06:43:04.866ZDEBGtest_snapshot_basic (collection-task): collected results from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 n_results = 1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
94762025-04-30T06:43:04.866ZTRACtest_snapshot_basic (collection-task): forwarded results to main collection loop collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
94772025-04-30T06:43:04.866ZTRACtest_snapshot_basic (collection-task): top of inner collection loop, waiting for next request collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
94782025-04-30T06:43:04.866ZDEBGtest_snapshot_basic (BackgroundTasks): client response SledAgent = http://[::1]:41923 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:41923/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "d676520d-1179-471f-b7cc-b6c8f0b21205", "content-length": "3141", "date": "Wed, 30 Apr 2025 06:43:04 GMT"} })
94792025-04-30T06:43:04.868ZDEBGtest_snapshot_basic (BackgroundTasks): begin collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
94802025-04-30T06:43:04.868ZDEBGtest_snapshot_basic (BackgroundTasks): end collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
94812025-04-30T06:43:04.868ZDEBGtest_snapshot_basic (BackgroundTasks): finished collection background_task = inventory_collection
94822025-04-30T06:43:04.870ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
94832025-04-30T06:43:04.870ZTRACtest_snapshot_basic (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
94842025-04-30T06:43:04.870ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
94852025-04-30T06:43:04.870ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
94862025-04-30T06:43:04.871ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
94872025-04-30T06:43:04.875ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
94882025-04-30T06:43:04.887ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
94892025-04-30T06:43:04.887ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
94902025-04-30T06:43:04.887ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
94912025-04-30T06:43:04.888ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
94922025-04-30T06:43:04.932ZDEBGtest_snapshot_basic: attempt to fetch latest inventory collection result = Ok("found")
94932025-04-30T06:43:04.942ZWARNtest_snapshot_basic (datastore): Non-retryable transaction failure err = DatabaseError(UniqueViolation, "duplicate key value violates unique constraint \\"inv_caboose_pkey\\"") transaction = inventory_insert_collection
94942025-04-30T06:43:04.943ZWARNtest_snapshot_basic (BackgroundTasks): inventory collection failed background_task = inventory_collection error = failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint "inv_caboose_pkey"
94952025-04-30T06:43:04.943ZDEBGtest_snapshot_basic (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 525.761114ms iteration = 32 status = {"error":"failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint \\"inv_caboose_pkey\\""}
94962025-04-30T06:43:04.943ZDEBGtest_snapshot_basic (BackgroundTasks): activating background_task = inventory_collection iteration = 33 reason = Signaled
94972025-04-30T06:43:04.943ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
94982025-04-30T06:43:04.943ZTRACtest_snapshot_basic (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
94992025-04-30T06:43:04.944ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
95002025-04-30T06:43:04.944ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
95012025-04-30T06:43:04.945ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
95022025-04-30T06:43:04.948ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
95032025-04-30T06:43:04.955ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
95042025-04-30T06:43:04.955ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
95052025-04-30T06:43:04.955ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
95062025-04-30T06:43:04.956ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
95072025-04-30T06:43:04.967ZDEBGtest_snapshot_basic (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = [(d1ad44e5-18dd-4193-a70e-bf9042f0e0e1, 0)]
95082025-04-30T06:43:04.967ZTRACtest_snapshot_basic (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
95092025-04-30T06:43:04.976ZTRACtest_snapshot_basic (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: f0 e8 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 ............._mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:55029 req_id = 91558533-37cd-4421-9bb5-d06e7f11bb22
95102025-04-30T06:43:04.976ZDEBGtest_snapshot_basic (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 61672,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:55029 req_id = 91558533-37cd-4421-9bb5-d06e7f11bb22
95112025-04-30T06:43:04.977ZDEBGtest_snapshot_basic (store): query key key = _mgs._tcp
95122025-04-30T06:43:04.977ZDEBGtest_snapshot_basic (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
95132025-04-30T06:43:04.977ZDEBGtest_snapshot_basic (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:55029 query = LowerQuery { name: LowerName(Name("_mgs._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 41502, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = 91558533-37cd-4421-9bb5-d06e7f11bb22
95142025-04-30T06:43:04.978ZDEBGtest_snapshot_basic (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 41502, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 4249, tv_nsec: 777228795 } })
95152025-04-30T06:43:04.978ZTRACtest_snapshot_basic (DnsResolver): lookup_service_targets: looking up SRV target
95162025-04-30T06:43:04.979ZTRACtest_snapshot_basic (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: 64 46 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e dF..........-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:59184 req_id = 8a95c95d-0064-451d-914b-e7ca3eda1d34
95172025-04-30T06:43:04.979ZDEBGtest_snapshot_basic (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 25670,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:59184 req_id = 8a95c95d-0064-451d-914b-e7ca3eda1d34
95182025-04-30T06:43:04.979ZDEBGtest_snapshot_basic (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
95192025-04-30T06:43:04.979ZDEBGtest_snapshot_basic (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:59184 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = 8a95c95d-0064-451d-914b-e7ca3eda1d34
95202025-04-30T06:43:05.098ZTRACtest_snapshot_basic (DnsResolver): lookup_all_socket_v6 srv dns_name = _clickhouse-admin-keeper._tcp.control-plane.oxide.internal
95212025-04-30T06:43:05.098ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
95222025-04-30T06:43:05.098ZTRACtest_snapshot_basic (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
95232025-04-30T06:43:05.098ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
95242025-04-30T06:43:05.098ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {} }
95252025-04-30T06:43:05.099ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
95262025-04-30T06:43:05.099ZDEBGtest_snapshot_basic (results-sink): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
95272025-04-30T06:43:05.099ZDEBGtest_snapshot_basic (results-sink-cluster): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
95282025-04-30T06:43:05.100ZDEBGtest_snapshot_basic (clickhouse-client): unrolling 1 total samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 6ad06c6b-9a6e-4ed1-b3f4-5d2e9da3f8a7
95292025-04-30T06:43:05.101ZTRACtest_snapshot_basic (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 6ad06c6b-9a6e-4ed1-b3f4-5d2e9da3f8a7 n_columns = 4 n_rows = 2 sql = INSERT INTO oximeter.fields_string FORMAT Native
95302025-04-30T06:43:05.101ZTRACtest_snapshot_basic (dns): buffer buffer = Length: 87 (0x57) bytes\n0000: e3 2b 01 00 00 01 00 00 00 00 00 01 18 5f 63 6c .+..........._cl\n0010: 69 63 6b 68 6f 75 73 65 2d 61 64 6d 69 6e 2d 6b ickhouse-admin-k\n0020: 65 65 70 65 72 04 5f 74 63 70 0d 63 6f 6e 74 72 eeper._tcp.contr\n0030: 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 69 ol-plane.oxide.i\n0040: 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 04 nternal..!....).\n0050: d0 00 00 00 00 00 00 ....... kind = dns peer_addr = [::1]:62907 req_id = 93dca23a-d3ef-4047-a75f-71ed5cc76913
95312025-04-30T06:43:05.101ZDEBGtest_snapshot_basic (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 58155,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 24,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 97,\n 100,\n 109,\n 105,\n 110,\n 45,\n 107,\n 101,\n 101,\n 112,\n 101,\n 114,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:62907 req_id = 93dca23a-d3ef-4047-a75f-71ed5cc76913
95322025-04-30T06:43:05.102ZDEBGtest_snapshot_basic (store): query key key = _clickhouse-admin-keeper._tcp
95332025-04-30T06:43:05.102ZERROtest_snapshot_basic (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 58155, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } NXDOMAIN: no records found for name: "_clickhouse-admin-keeper._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:62907 req_id = 93dca23a-d3ef-4047-a75f-71ed5cc76913
95342025-04-30T06:43:05.102ZDEBGtest_snapshot_basic (BackgroundTasks): begin collection background_task = inventory_collection
95352025-04-30T06:43:05.102ZDEBGtest_snapshot_basic (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:41502
95362025-04-30T06:43:05.102ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/ignition
95372025-04-30T06:43:05.103ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
95382025-04-30T06:43:05.109ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
95392025-04-30T06:43:05.109ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
95402025-04-30T06:43:05.109ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {} }
95412025-04-30T06:43:05.110ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
95422025-04-30T06:43:05.113ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
95432025-04-30T06:43:05.113ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {} }
95442025-04-30T06:43:05.113ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
95452025-04-30T06:43:05.114ZDEBGtest_snapshot_basic (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 6ad06c6b-9a6e-4ed1-b3f4-5d2e9da3f8a7 n_rows = 2 table_name = fields_string
95462025-04-30T06:43:05.114ZTRACtest_snapshot_basic (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 6ad06c6b-9a6e-4ed1-b3f4-5d2e9da3f8a7 n_columns = 4 n_rows = 1 sql = INSERT INTO oximeter.measurements_i64 FORMAT Native
95472025-04-30T06:43:05.115ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "f537d1eb-2e91-444f-968c-ff42aa57e807", "content-length": "835", "date": "Wed, 30 Apr 2025 06:43:04 GMT"} })
95482025-04-30T06:43:05.116ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/0
95492025-04-30T06:43:05.118ZDEBGtest_snapshot_basic (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 6ad06c6b-9a6e-4ed1-b3f4-5d2e9da3f8a7 n_rows = 1 table_name = measurements_i64
95502025-04-30T06:43:05.118ZTRACtest_snapshot_basic (results-sink): successfully inserted samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
95512025-04-30T06:43:05.119ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "82afe183-7a6e-4d16-96bb-f513553fb729", "content-length": "734", "date": "Wed, 30 Apr 2025 06:43:04 GMT"} })
95522025-04-30T06:43:05.119ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/0/component/sp/caboose?firmware_slot=0
95532025-04-30T06:43:05.121ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "42e01fa1-ed6a-4b65-a810-457d390878fe", "content-length": "111", "date": "Wed, 30 Apr 2025 06:43:04 GMT"} })
95542025-04-30T06:43:05.121ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/0/component/sp/caboose?firmware_slot=1
95552025-04-30T06:43:05.125ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "dd87f766-b51c-4eb6-8517-3fc149262402", "content-length": "111", "date": "Wed, 30 Apr 2025 06:43:04 GMT"} })
95562025-04-30T06:43:05.125ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/0/component/rot/caboose?firmware_slot=0
95572025-04-30T06:43:05.132ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "a48b6d2f-89c0-4d80-b9c2-9eec0792bf74", "content-length": "170", "date": "Wed, 30 Apr 2025 06:43:04 GMT"} })
95582025-04-30T06:43:05.132ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/0/component/rot/caboose?firmware_slot=1
95592025-04-30T06:43:05.136ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "52e0f794-5604-40d3-b12e-2293386fee60", "content-length": "170", "date": "Wed, 30 Apr 2025 06:43:04 GMT"} })
95602025-04-30T06:43:05.136ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/0/component/stage0/caboose?firmware_slot=0
95612025-04-30T06:43:05.141ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "61362013-24dc-4f17-b0d6-3f9cebd649cc", "content-length": "179", "date": "Wed, 30 Apr 2025 06:43:04 GMT"} })
95622025-04-30T06:43:05.141ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/0/component/stage0/caboose?firmware_slot=1
95632025-04-30T06:43:05.147ZDEBGtest_snapshot_basic: attempt to fetch latest inventory collection result = Ok("found")
95642025-04-30T06:43:05.148ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "7889c1c3-a318-4283-ab3d-6de666e28a61", "content-length": "179", "date": "Wed, 30 Apr 2025 06:43:04 GMT"} })
95652025-04-30T06:43:05.149ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/0/component/rot/cmpa
95662025-04-30T06:43:05.151ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "914f605f-03a3-41c1-9c2d-906475fcd5f7", "content-length": "702", "date": "Wed, 30 Apr 2025 06:43:04 GMT"} })
95672025-04-30T06:43:05.152ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/0/component/rot/cfpa
95682025-04-30T06:43:05.153ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "d6df3306-809e-4ac4-a786-cda40544a597", "content-length": "727", "date": "Wed, 30 Apr 2025 06:43:04 GMT"} })
95692025-04-30T06:43:05.153ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/0/component/rot/cfpa
95702025-04-30T06:43:05.155ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "e852994f-39a9-4c14-ab95-81d2ca4635f1", "content-length": "729", "date": "Wed, 30 Apr 2025 06:43:04 GMT"} })
95712025-04-30T06:43:05.156ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/0/component/rot/cfpa
95722025-04-30T06:43:05.157ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "9234d4a1-3550-4639-82bf-b6a85c0fb6fc", "content-length": "728", "date": "Wed, 30 Apr 2025 06:43:04 GMT"} })
95732025-04-30T06:43:05.157ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/1
95742025-04-30T06:43:05.159ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "937390a2-bfee-4c09-90f2-c22a6518a615", "content-length": "734", "date": "Wed, 30 Apr 2025 06:43:04 GMT"} })
95752025-04-30T06:43:05.159ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/1/component/sp/caboose?firmware_slot=0
95762025-04-30T06:43:05.162ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "32c9a2b1-10ef-413f-9e4c-b6e1234ac505", "content-length": "111", "date": "Wed, 30 Apr 2025 06:43:04 GMT"} })
95772025-04-30T06:43:05.162ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/1/component/sp/caboose?firmware_slot=1
95782025-04-30T06:43:05.169ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "58a538a9-460e-4b1d-a3aa-30818c06be5e", "content-length": "111", "date": "Wed, 30 Apr 2025 06:43:04 GMT"} })
95792025-04-30T06:43:05.169ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/1/component/rot/caboose?firmware_slot=0
95802025-04-30T06:43:05.175ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "310b324c-efbd-47aa-9f09-0058ecc38663", "content-length": "170", "date": "Wed, 30 Apr 2025 06:43:04 GMT"} })
95812025-04-30T06:43:05.175ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/1/component/rot/caboose?firmware_slot=1
95822025-04-30T06:43:05.177ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "7ea6071a-323a-42e7-8b91-600fb4721280", "content-length": "170", "date": "Wed, 30 Apr 2025 06:43:04 GMT"} })
95832025-04-30T06:43:05.178ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/1/component/stage0/caboose?firmware_slot=0
95842025-04-30T06:43:05.179ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "dfa4690f-56ec-4074-8a2a-c863e09eef2e", "content-length": "179", "date": "Wed, 30 Apr 2025 06:43:04 GMT"} })
95852025-04-30T06:43:05.187ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/1/component/stage0/caboose?firmware_slot=1
95862025-04-30T06:43:05.193ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "7de6f928-47a9-4d1b-a04b-41ce6ba548f0", "content-length": "179", "date": "Wed, 30 Apr 2025 06:43:04 GMT"} })
95872025-04-30T06:43:05.193ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/1/component/rot/cmpa
95882025-04-30T06:43:05.194ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "c428bdee-28ad-4bee-9142-740ae08bbc10", "content-length": "702", "date": "Wed, 30 Apr 2025 06:43:04 GMT"} })
95892025-04-30T06:43:05.195ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/1/component/rot/cfpa
95902025-04-30T06:43:05.196ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "5331c297-8103-41ea-9cd6-82a4c1a407f8", "content-length": "727", "date": "Wed, 30 Apr 2025 06:43:04 GMT"} })
95912025-04-30T06:43:05.196ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/1/component/rot/cfpa
95922025-04-30T06:43:05.197ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "628e720f-1041-4f9d-8f34-343f3c21047d", "content-length": "729", "date": "Wed, 30 Apr 2025 06:43:04 GMT"} })
95932025-04-30T06:43:05.198ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/1/component/rot/cfpa
95942025-04-30T06:43:05.199ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
95952025-04-30T06:43:05.199ZTRACtest_snapshot_basic (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
95962025-04-30T06:43:05.199ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
95972025-04-30T06:43:05.199ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {} }
95982025-04-30T06:43:05.200ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
95992025-04-30T06:43:05.200ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "61b4ba9e-c665-4fe5-bf40-d4fbfeb0f9f7", "content-length": "728", "date": "Wed, 30 Apr 2025 06:43:04 GMT"} })
96002025-04-30T06:43:05.200ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/0
96012025-04-30T06:43:05.201ZWARNtest_snapshot_basic (results-sink-cluster): failed to insert some results into metric DB collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 error = Failed to check out connection to database: No backends found for this service file = oximeter/collector/src/results_sink.rs:92
96022025-04-30T06:43:05.205ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "3659fe94-e03a-4dd7-a067-684c37c83529", "content-length": "723", "date": "Wed, 30 Apr 2025 06:43:04 GMT"} })
96032025-04-30T06:43:05.205ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/0/component/sp/caboose?firmware_slot=0
96042025-04-30T06:43:05.205ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
96052025-04-30T06:43:05.210ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
96062025-04-30T06:43:05.210ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
96072025-04-30T06:43:05.210ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {} }
96082025-04-30T06:43:05.211ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
96092025-04-30T06:43:05.216ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
96102025-04-30T06:43:05.216ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {} }
96112025-04-30T06:43:05.217ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
96122025-04-30T06:43:05.219ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "2592e890-6cf1-40e9-a535-ce36575540ea", "content-length": "109", "date": "Wed, 30 Apr 2025 06:43:04 GMT"} })
96132025-04-30T06:43:05.219ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/0/component/sp/caboose?firmware_slot=1
96142025-04-30T06:43:05.224ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "96a39f91-4418-4b3f-9fd9-511e95f8a0b0", "content-length": "109", "date": "Wed, 30 Apr 2025 06:43:04 GMT"} })
96152025-04-30T06:43:05.227ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/0/component/rot/caboose?firmware_slot=0
96162025-04-30T06:43:05.230ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "6c733991-035c-42f7-9cf2-1fa57eae273b", "content-length": "169", "date": "Wed, 30 Apr 2025 06:43:04 GMT"} })
96172025-04-30T06:43:05.230ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/0/component/rot/caboose?firmware_slot=1
96182025-04-30T06:43:05.232ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "95c9043d-2a2e-43ba-a2dc-1f1b73e1d00c", "content-length": "169", "date": "Wed, 30 Apr 2025 06:43:04 GMT"} })
96192025-04-30T06:43:05.232ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/0/component/stage0/caboose?firmware_slot=0
96202025-04-30T06:43:05.235ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "f9711e51-1935-4eef-9020-ffbb0697e3f2", "content-length": "178", "date": "Wed, 30 Apr 2025 06:43:04 GMT"} })
96212025-04-30T06:43:05.235ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/0/component/stage0/caboose?firmware_slot=1
96222025-04-30T06:43:05.237ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "5e070a58-5f1e-4dcb-95db-20d67e20b6b4", "content-length": "178", "date": "Wed, 30 Apr 2025 06:43:04 GMT"} })
96232025-04-30T06:43:05.238ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/0/component/rot/cmpa
96242025-04-30T06:43:05.239ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "73825d73-4e36-4152-89ff-1c4ceaa4b381", "content-length": "702", "date": "Wed, 30 Apr 2025 06:43:04 GMT"} })
96252025-04-30T06:43:05.244ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/0/component/rot/cfpa
96262025-04-30T06:43:05.256ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "82ef9038-e157-44a2-8148-98bc253ccf9e", "content-length": "727", "date": "Wed, 30 Apr 2025 06:43:04 GMT"} })
96272025-04-30T06:43:05.256ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/0/component/rot/cfpa
96282025-04-30T06:43:05.259ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "39592470-2575-45bf-92d2-b83573d6915e", "content-length": "729", "date": "Wed, 30 Apr 2025 06:43:04 GMT"} })
96292025-04-30T06:43:05.263ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/0/component/rot/cfpa
96302025-04-30T06:43:05.266ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "e546eff6-049d-467e-acb7-7814062738ad", "content-length": "728", "date": "Wed, 30 Apr 2025 06:43:04 GMT"} })
96312025-04-30T06:43:05.266ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/1
96322025-04-30T06:43:05.277ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "91b78738-ab6a-4b4c-b66f-55734cd1ea30", "content-length": "723", "date": "Wed, 30 Apr 2025 06:43:04 GMT"} })
96332025-04-30T06:43:05.277ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/1/component/sp/caboose?firmware_slot=0
96342025-04-30T06:43:05.277ZDEBGtest_snapshot_basic: attempt to fetch latest inventory collection result = Ok("found")
96352025-04-30T06:43:05.284ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "7474f281-cff8-4ae4-8188-308b4bfcb4d5", "content-length": "109", "date": "Wed, 30 Apr 2025 06:43:04 GMT"} })
96362025-04-30T06:43:05.285ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/1/component/sp/caboose?firmware_slot=1
96372025-04-30T06:43:05.289ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "8e5cce28-197b-46ad-9702-6ac72577ad7d", "content-length": "109", "date": "Wed, 30 Apr 2025 06:43:04 GMT"} })
96382025-04-30T06:43:05.289ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/1/component/rot/caboose?firmware_slot=0
96392025-04-30T06:43:05.292ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "966fec35-37ed-4738-a838-09e965d39e9e", "content-length": "169", "date": "Wed, 30 Apr 2025 06:43:04 GMT"} })
96402025-04-30T06:43:05.292ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/1/component/rot/caboose?firmware_slot=1
96412025-04-30T06:43:05.294ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "caacf204-ed00-48b4-a4b9-d0630b9dd4e4", "content-length": "169", "date": "Wed, 30 Apr 2025 06:43:04 GMT"} })
96422025-04-30T06:43:05.294ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/1/component/stage0/caboose?firmware_slot=0
96432025-04-30T06:43:05.297ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "dda4ce34-d709-44e6-b8bd-793a5f8e4fa6", "content-length": "178", "date": "Wed, 30 Apr 2025 06:43:04 GMT"} })
96442025-04-30T06:43:05.298ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/1/component/stage0/caboose?firmware_slot=1
96452025-04-30T06:43:05.299ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "6b352f9a-8308-4348-aad7-0b835eeeb0e2", "content-length": "178", "date": "Wed, 30 Apr 2025 06:43:04 GMT"} })
96462025-04-30T06:43:05.299ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/1/component/rot/cmpa
96472025-04-30T06:43:05.300ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "be34820f-f949-4e4f-8ce5-4c31985da603", "content-length": "702", "date": "Wed, 30 Apr 2025 06:43:04 GMT"} })
96482025-04-30T06:43:05.300ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/1/component/rot/cfpa
96492025-04-30T06:43:05.305ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "aa191e75-c204-4d73-af0c-5473b90f98ab", "content-length": "727", "date": "Wed, 30 Apr 2025 06:43:04 GMT"} })
96502025-04-30T06:43:05.306ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/1/component/rot/cfpa
96512025-04-30T06:43:05.307ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "cf9e11ab-3563-4229-b02c-60499bc4760a", "content-length": "729", "date": "Wed, 30 Apr 2025 06:43:04 GMT"} })
96522025-04-30T06:43:05.307ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/1/component/rot/cfpa
96532025-04-30T06:43:05.308ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "0c9dab86-6a09-4814-b933-951e4227e463", "content-length": "728", "date": "Wed, 30 Apr 2025 06:43:04 GMT"} })
96542025-04-30T06:43:05.308ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
96552025-04-30T06:43:05.308ZTRACtest_snapshot_basic (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
96562025-04-30T06:43:05.308ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
96572025-04-30T06:43:05.308ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
96582025-04-30T06:43:05.309ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
96592025-04-30T06:43:05.313ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
96602025-04-30T06:43:05.316ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
96612025-04-30T06:43:05.316ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
96622025-04-30T06:43:05.316ZTRACtest_snapshot_basic (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
96632025-04-30T06:43:05.316ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
96642025-04-30T06:43:05.316ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
96652025-04-30T06:43:05.317ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
96662025-04-30T06:43:05.323ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
96672025-04-30T06:43:05.327ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
96682025-04-30T06:43:05.328ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
96692025-04-30T06:43:05.328ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
96702025-04-30T06:43:05.328ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
96712025-04-30T06:43:05.330ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
96722025-04-30T06:43:05.330ZTRACtest_snapshot_basic (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
96732025-04-30T06:43:05.330ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
96742025-04-30T06:43:05.330ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {} }
96752025-04-30T06:43:05.331ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
96762025-04-30T06:43:05.409ZDEBGtest_snapshot_basic (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:41923
96772025-04-30T06:43:05.409ZDEBGtest_snapshot_basic (BackgroundTasks): client request SledAgent = http://[::1]:41923 background_task = inventory_collection body = None method = GET uri = http://[::1]:41923/inventory
96782025-04-30T06:43:05.427ZINFOtest_snapshot_basic (dropshot): accepted connection local_addr = [::1]:41923 remote_addr = [::1]:57599 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
96792025-04-30T06:43:05.427ZTRACtest_snapshot_basic (dropshot): incoming request local_addr = [::1]:41923 method = GET remote_addr = [::1]:57599 req_id = 56fb799a-a809-4340-8893-bd814787f70b sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
96802025-04-30T06:43:05.428ZINFOtest_snapshot_basic (dropshot): request completed latency_us = 651 local_addr = [::1]:41923 method = GET remote_addr = [::1]:57599 req_id = 56fb799a-a809-4340-8893-bd814787f70b response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
96812025-04-30T06:43:05.429ZDEBGtest_snapshot_basic (BackgroundTasks): client response SledAgent = http://[::1]:41923 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:41923/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "56fb799a-a809-4340-8893-bd814787f70b", "content-length": "3141", "date": "Wed, 30 Apr 2025 06:43:04 GMT"} })
96822025-04-30T06:43:05.431ZDEBGtest_snapshot_basic (BackgroundTasks): begin collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
96832025-04-30T06:43:05.431ZDEBGtest_snapshot_basic (BackgroundTasks): end collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
96842025-04-30T06:43:05.431ZDEBGtest_snapshot_basic (BackgroundTasks): finished collection background_task = inventory_collection
96852025-04-30T06:43:05.433ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
96862025-04-30T06:43:05.433ZTRACtest_snapshot_basic (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
96872025-04-30T06:43:05.434ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
96882025-04-30T06:43:05.434ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
96892025-04-30T06:43:05.435ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
96902025-04-30T06:43:05.435ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
96912025-04-30T06:43:05.444ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
96922025-04-30T06:43:05.444ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
96932025-04-30T06:43:05.444ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {} }
96942025-04-30T06:43:05.445ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
96952025-04-30T06:43:05.447ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
96962025-04-30T06:43:05.456ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
96972025-04-30T06:43:05.457ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
96982025-04-30T06:43:05.457ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
96992025-04-30T06:43:05.457ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
97002025-04-30T06:43:05.460ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
97012025-04-30T06:43:05.460ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {} }
97022025-04-30T06:43:05.461ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
97032025-04-30T06:43:05.534ZWARNtest_snapshot_basic (datastore): Non-retryable transaction failure err = DatabaseError(UniqueViolation, "duplicate key value violates unique constraint \\"inv_caboose_pkey\\"") transaction = inventory_insert_collection
97042025-04-30T06:43:05.534ZWARNtest_snapshot_basic (BackgroundTasks): inventory collection failed background_task = inventory_collection error = failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint "inv_caboose_pkey"
97052025-04-30T06:43:05.534ZDEBGtest_snapshot_basic (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 590.868687ms iteration = 33 status = {"error":"failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint \\"inv_caboose_pkey\\""}
97062025-04-30T06:43:05.534ZDEBGtest_snapshot_basic (BackgroundTasks): activating background_task = inventory_collection iteration = 34 reason = Signaled
97072025-04-30T06:43:05.535ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
97082025-04-30T06:43:05.535ZTRACtest_snapshot_basic (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
97092025-04-30T06:43:05.535ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
97102025-04-30T06:43:05.535ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
97112025-04-30T06:43:05.536ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
97122025-04-30T06:43:05.541ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
97132025-04-30T06:43:05.547ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
97142025-04-30T06:43:05.547ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
97152025-04-30T06:43:05.547ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
97162025-04-30T06:43:05.548ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
97172025-04-30T06:43:05.567ZDEBGtest_snapshot_basic (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = [(d1ad44e5-18dd-4193-a70e-bf9042f0e0e1, 0)]
97182025-04-30T06:43:05.567ZTRACtest_snapshot_basic (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
97192025-04-30T06:43:05.572ZTRACtest_snapshot_basic (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: cc c4 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 ............._mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:53659 req_id = 0c2b2449-671a-4c79-a7f3-2b856e5fd775
97202025-04-30T06:43:05.572ZDEBGtest_snapshot_basic (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 52420,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:53659 req_id = 0c2b2449-671a-4c79-a7f3-2b856e5fd775
97212025-04-30T06:43:05.573ZDEBGtest_snapshot_basic (store): query key key = _mgs._tcp
97222025-04-30T06:43:05.573ZDEBGtest_snapshot_basic (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
97232025-04-30T06:43:05.573ZDEBGtest_snapshot_basic (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:53659 query = LowerQuery { name: LowerName(Name("_mgs._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 41502, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = 0c2b2449-671a-4c79-a7f3-2b856e5fd775
97242025-04-30T06:43:05.573ZDEBGtest_snapshot_basic: attempt to fetch latest inventory collection result = Ok("found")
97252025-04-30T06:43:05.574ZDEBGtest_snapshot_basic (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 41502, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 4250, tv_nsec: 373341180 } })
97262025-04-30T06:43:05.574ZTRACtest_snapshot_basic (DnsResolver): lookup_service_targets: looking up SRV target
97272025-04-30T06:43:05.584ZTRACtest_snapshot_basic (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: 1a 00 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e ............-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:59781 req_id = 2f36b0e4-bead-4275-b379-e621a76a08db
97282025-04-30T06:43:05.585ZDEBGtest_snapshot_basic (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 6656,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:59781 req_id = 2f36b0e4-bead-4275-b379-e621a76a08db
97292025-04-30T06:43:05.586ZDEBGtest_snapshot_basic (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
97302025-04-30T06:43:05.587ZDEBGtest_snapshot_basic (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:59781 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = 2f36b0e4-bead-4275-b379-e621a76a08db
97312025-04-30T06:43:05.721ZTRACtest_snapshot_basic (DnsResolver): lookup_all_socket_v6 srv dns_name = _clickhouse-admin-keeper._tcp.control-plane.oxide.internal
97322025-04-30T06:43:05.722ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
97332025-04-30T06:43:05.722ZTRACtest_snapshot_basic (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
97342025-04-30T06:43:05.722ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
97352025-04-30T06:43:05.722ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {} }
97362025-04-30T06:43:05.723ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
97372025-04-30T06:43:05.724ZINFOtest_snapshot_basic (oximeter-agent): refreshing list of producers from Nexus collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 file = oximeter/collector/src/agent.rs:562
97382025-04-30T06:43:05.727ZDEBGtest_snapshot_basic (oximeter): client request body = None method = GET uri = http://127.0.0.1:61468/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
97392025-04-30T06:43:05.730ZTRACtest_snapshot_basic (dropshot_internal): incoming request local_addr = 127.0.0.1:61468 method = GET remote_addr = 127.0.0.1:64332 req_id = 9808c445-a0c2-4025-a2e8-ac0d340a0642 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
97402025-04-30T06:43:05.730ZTRACtest_snapshot_basic (dns): buffer buffer = Length: 87 (0x57) bytes\n0000: 41 3a 01 00 00 01 00 00 00 00 00 01 18 5f 63 6c A:..........._cl\n0010: 69 63 6b 68 6f 75 73 65 2d 61 64 6d 69 6e 2d 6b ickhouse-admin-k\n0020: 65 65 70 65 72 04 5f 74 63 70 0d 63 6f 6e 74 72 eeper._tcp.contr\n0030: 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 69 ol-plane.oxide.i\n0040: 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 04 nternal..!....).\n0050: d0 00 00 00 00 00 00 ....... kind = dns peer_addr = [::1]:56758 req_id = 563b6e85-8152-4707-b2cc-0e628115c051
97412025-04-30T06:43:05.730ZDEBGtest_snapshot_basic (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 16698,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 24,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 97,\n 100,\n 109,\n 105,\n 110,\n 45,\n 107,\n 101,\n 101,\n 112,\n 101,\n 114,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:56758 req_id = 563b6e85-8152-4707-b2cc-0e628115c051
97422025-04-30T06:43:05.730ZDEBGtest_snapshot_basic (store): query key key = _clickhouse-admin-keeper._tcp
97432025-04-30T06:43:05.730ZERROtest_snapshot_basic (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 16698, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } NXDOMAIN: no records found for name: "_clickhouse-admin-keeper._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:56758 req_id = 563b6e85-8152-4707-b2cc-0e628115c051
97442025-04-30T06:43:05.731ZTRACtest_snapshot_basic (dropshot_internal): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:61468 method = GET remote_addr = 127.0.0.1:64332 req_id = 9808c445-a0c2-4025-a2e8-ac0d340a0642 resource = Database uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
97452025-04-30T06:43:05.731ZDEBGtest_snapshot_basic (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:61468 method = GET remote_addr = 127.0.0.1:64332 req_id = 9808c445-a0c2-4025-a2e8-ac0d340a0642 roles = RoleSet { roles: {} } uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
97462025-04-30T06:43:05.732ZDEBGtest_snapshot_basic (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:61468 method = GET remote_addr = 127.0.0.1:64332 req_id = 9808c445-a0c2-4025-a2e8-ac0d340a0642 resource = Database result = Ok(()) uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
97472025-04-30T06:43:05.733ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
97482025-04-30T06:43:05.745ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
97492025-04-30T06:43:05.746ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
97502025-04-30T06:43:05.747ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {} }
97512025-04-30T06:43:05.747ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
97522025-04-30T06:43:05.748ZDEBGtest_snapshot_basic (BackgroundTasks): begin collection background_task = inventory_collection
97532025-04-30T06:43:05.748ZDEBGtest_snapshot_basic (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:41502
97542025-04-30T06:43:05.751ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/ignition
97552025-04-30T06:43:05.755ZINFOtest_snapshot_basic (dropshot_internal): request completed latency_us = 25160 local_addr = 127.0.0.1:61468 method = GET remote_addr = 127.0.0.1:64332 req_id = 9808c445-a0c2-4025-a2e8-ac0d340a0642 response_code = 200 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
97562025-04-30T06:43:05.759ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
97572025-04-30T06:43:05.767ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {} }
97582025-04-30T06:43:05.769ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
97592025-04-30T06:43:05.771ZDEBGtest_snapshot_basic (oximeter): client response result = Ok(Response { url: "http://127.0.0.1:61468/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending", status: 200, headers: {"content-type": "application/json", "x-request-id": "9808c445-a0c2-4025-a2e8-ac0d340a0642", "content-length": "405", "date": "Wed, 30 Apr 2025 06:43:05 GMT"} })
97602025-04-30T06:43:05.771ZDEBGtest_snapshot_basic (oximeter): client request body = None method = GET uri = http://127.0.0.1:61468/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
97612025-04-30T06:43:05.771ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "ff5972b7-86c4-44c9-ae20-28e06233dd87", "content-length": "835", "date": "Wed, 30 Apr 2025 06:43:05 GMT"} })
97622025-04-30T06:43:05.772ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/0
97632025-04-30T06:43:05.773ZTRACtest_snapshot_basic (dropshot_internal): incoming request local_addr = 127.0.0.1:61468 method = GET remote_addr = 127.0.0.1:64332 req_id = 93df93c6-0fab-462b-9e1d-3019baa7ceb0 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
97642025-04-30T06:43:05.773ZTRACtest_snapshot_basic (dropshot_internal): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:61468 method = GET remote_addr = 127.0.0.1:64332 req_id = 93df93c6-0fab-462b-9e1d-3019baa7ceb0 resource = Database uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
97652025-04-30T06:43:05.773ZDEBGtest_snapshot_basic (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:61468 method = GET remote_addr = 127.0.0.1:64332 req_id = 93df93c6-0fab-462b-9e1d-3019baa7ceb0 roles = RoleSet { roles: {} } uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
97662025-04-30T06:43:05.774ZDEBGtest_snapshot_basic (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:61468 method = GET remote_addr = 127.0.0.1:64332 req_id = 93df93c6-0fab-462b-9e1d-3019baa7ceb0 resource = Database result = Ok(()) uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
97672025-04-30T06:43:05.775ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "1348a75a-c0b6-47dd-9cea-5f957c14d5dc", "content-length": "734", "date": "Wed, 30 Apr 2025 06:43:05 GMT"} })
97682025-04-30T06:43:05.776ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/0/component/sp/caboose?firmware_slot=0
97692025-04-30T06:43:05.777ZINFOtest_snapshot_basic (dropshot_internal): request completed latency_us = 4126 local_addr = 127.0.0.1:61468 method = GET remote_addr = 127.0.0.1:64332 req_id = 93df93c6-0fab-462b-9e1d-3019baa7ceb0 response_code = 200 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
97702025-04-30T06:43:05.779ZDEBGtest_snapshot_basic (oximeter): client response result = Ok(Response { url: "http://127.0.0.1:61468/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D", status: 200, headers: {"content-type": "application/json", "x-request-id": "93df93c6-0fab-462b-9e1d-3019baa7ceb0", "content-length": "29", "date": "Wed, 30 Apr 2025 06:43:05 GMT"} })
97712025-04-30T06:43:05.779ZTRACtest_snapshot_basic (oximeter-agent): ignoring request to update existing metric producer, since the endpoint information is the same as the existing collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = 354a52d4-3078-4706-81ce-12c32f6bbeaa
97722025-04-30T06:43:05.779ZTRACtest_snapshot_basic (oximeter-agent): ignoring request to update existing metric producer, since the endpoint information is the same as the existing collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
97732025-04-30T06:43:05.779ZINFOtest_snapshot_basic (oximeter-agent): refreshed list of producers from Nexus collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 file = oximeter/collector/src/agent.rs:637 n_current_tasks = 2 n_pruned_tasks = 0
97742025-04-30T06:43:05.782ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "530c091b-8340-40f5-80ae-d22e867723ac", "content-length": "111", "date": "Wed, 30 Apr 2025 06:43:05 GMT"} })
97752025-04-30T06:43:05.782ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/0/component/sp/caboose?firmware_slot=1
97762025-04-30T06:43:05.785ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "b6175fc8-5fdb-474b-bf7c-b50adab2467c", "content-length": "111", "date": "Wed, 30 Apr 2025 06:43:05 GMT"} })
97772025-04-30T06:43:05.785ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/0/component/rot/caboose?firmware_slot=0
97782025-04-30T06:43:05.787ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "9af4e516-e680-47f4-b71b-9bf5ee941ed5", "content-length": "170", "date": "Wed, 30 Apr 2025 06:43:05 GMT"} })
97792025-04-30T06:43:05.787ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/0/component/rot/caboose?firmware_slot=1
97802025-04-30T06:43:05.789ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "06476f64-0d52-4cc8-8c73-bb5f7231bd55", "content-length": "170", "date": "Wed, 30 Apr 2025 06:43:05 GMT"} })
97812025-04-30T06:43:05.789ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/0/component/stage0/caboose?firmware_slot=0
97822025-04-30T06:43:05.791ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "22d85fbf-1275-4c12-b03f-e971070a53b0", "content-length": "179", "date": "Wed, 30 Apr 2025 06:43:05 GMT"} })
97832025-04-30T06:43:05.791ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/0/component/stage0/caboose?firmware_slot=1
97842025-04-30T06:43:05.793ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "d52a969e-ece1-4838-a62c-b4e6d6383d5a", "content-length": "179", "date": "Wed, 30 Apr 2025 06:43:05 GMT"} })
97852025-04-30T06:43:05.793ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/0/component/rot/cmpa
97862025-04-30T06:43:05.795ZDEBGtest_snapshot_basic (collection-task): sent timer-based collection request to the collection task collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
97872025-04-30T06:43:05.795ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "0daa9a89-7e3a-4278-bece-a8abb1923e06", "content-length": "702", "date": "Wed, 30 Apr 2025 06:43:05 GMT"} })
97882025-04-30T06:43:05.795ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/0/component/rot/cfpa
97892025-04-30T06:43:05.795ZDEBGtest_snapshot_basic (collection-task): collecting from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
97902025-04-30T06:43:05.797ZTRACtest_snapshot_basic (collection-task): sent collection request to producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
97912025-04-30T06:43:05.798ZDEBGtest_snapshot_basic (collection-task): collected results from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 n_results = 1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
97922025-04-30T06:43:05.798ZTRACtest_snapshot_basic (collection-task): forwarded results to main collection loop collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
97932025-04-30T06:43:05.798ZTRACtest_snapshot_basic (collection-task): top of inner collection loop, waiting for next request collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
97942025-04-30T06:43:05.799ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "3638b3e0-0ff9-4a7f-84b0-bb13816c1805", "content-length": "727", "date": "Wed, 30 Apr 2025 06:43:05 GMT"} })
97952025-04-30T06:43:05.799ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/0/component/rot/cfpa
97962025-04-30T06:43:05.801ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "e7aac227-b1ee-4956-bcfe-fc959240155d", "content-length": "729", "date": "Wed, 30 Apr 2025 06:43:05 GMT"} })
97972025-04-30T06:43:05.801ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/0/component/rot/cfpa
97982025-04-30T06:43:05.802ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "055848d6-80ad-4f29-af4c-6d35b255b76b", "content-length": "728", "date": "Wed, 30 Apr 2025 06:43:05 GMT"} })
97992025-04-30T06:43:05.803ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/1
98002025-04-30T06:43:05.806ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "8620af97-35a0-423b-a927-bbbbeac395ec", "content-length": "734", "date": "Wed, 30 Apr 2025 06:43:05 GMT"} })
98012025-04-30T06:43:05.806ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/1/component/sp/caboose?firmware_slot=0
98022025-04-30T06:43:05.809ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "917d98f3-e5f1-41c2-8a5e-5fcd7b99436b", "content-length": "111", "date": "Wed, 30 Apr 2025 06:43:05 GMT"} })
98032025-04-30T06:43:05.809ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/1/component/sp/caboose?firmware_slot=1
98042025-04-30T06:43:05.814ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "f2472f28-fe4b-48ef-8106-38fd51692e5f", "content-length": "111", "date": "Wed, 30 Apr 2025 06:43:05 GMT"} })
98052025-04-30T06:43:05.814ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/1/component/rot/caboose?firmware_slot=0
98062025-04-30T06:43:05.815ZDEBGtest_snapshot_basic: attempt to fetch latest inventory collection result = Ok("found")
98072025-04-30T06:43:05.817ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "bbbb604b-1839-406c-ac09-8b191e9081d1", "content-length": "170", "date": "Wed, 30 Apr 2025 06:43:05 GMT"} })
98082025-04-30T06:43:05.817ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/1/component/rot/caboose?firmware_slot=1
98092025-04-30T06:43:05.819ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "1747af90-fb1d-4e47-a4a4-bea67c23c8e3", "content-length": "170", "date": "Wed, 30 Apr 2025 06:43:05 GMT"} })
98102025-04-30T06:43:05.820ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/1/component/stage0/caboose?firmware_slot=0
98112025-04-30T06:43:05.823ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "acbd9951-896e-4b4a-89e7-6f0f40dbf70b", "content-length": "179", "date": "Wed, 30 Apr 2025 06:43:05 GMT"} })
98122025-04-30T06:43:05.824ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/1/component/stage0/caboose?firmware_slot=1
98132025-04-30T06:43:05.826ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "2c652d49-0792-43b0-928d-e8a62bf444b1", "content-length": "179", "date": "Wed, 30 Apr 2025 06:43:05 GMT"} })
98142025-04-30T06:43:05.826ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/1/component/rot/cmpa
98152025-04-30T06:43:05.827ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "1ff94e4f-01a8-445b-95a3-1bd20dc06212", "content-length": "702", "date": "Wed, 30 Apr 2025 06:43:05 GMT"} })
98162025-04-30T06:43:05.827ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/1/component/rot/cfpa
98172025-04-30T06:43:05.831ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "cd402d02-1889-41b7-84bb-5a1222354fc3", "content-length": "727", "date": "Wed, 30 Apr 2025 06:43:05 GMT"} })
98182025-04-30T06:43:05.831ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/1/component/rot/cfpa
98192025-04-30T06:43:05.831ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "bf50401a-e87d-4065-8aeb-d58ed9d3ea3e", "content-length": "729", "date": "Wed, 30 Apr 2025 06:43:05 GMT"} })
98202025-04-30T06:43:05.831ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/1/component/rot/cfpa
98212025-04-30T06:43:05.832ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "21ae02f1-0c3e-49e5-b616-1c44dd8d84cb", "content-length": "728", "date": "Wed, 30 Apr 2025 06:43:05 GMT"} })
98222025-04-30T06:43:05.832ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/0
98232025-04-30T06:43:05.835ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "fad2d1c4-d552-434f-8f29-7ac8c50c56ec", "content-length": "723", "date": "Wed, 30 Apr 2025 06:43:05 GMT"} })
98242025-04-30T06:43:05.835ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/0/component/sp/caboose?firmware_slot=0
98252025-04-30T06:43:05.858ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "52d70a54-f63f-4c65-a362-b13ab22a5383", "content-length": "109", "date": "Wed, 30 Apr 2025 06:43:05 GMT"} })
98262025-04-30T06:43:05.858ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/0/component/sp/caboose?firmware_slot=1
98272025-04-30T06:43:05.862ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "bc8ae50e-11d9-4afd-9515-8e45d97badad", "content-length": "109", "date": "Wed, 30 Apr 2025 06:43:05 GMT"} })
98282025-04-30T06:43:05.862ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/0/component/rot/caboose?firmware_slot=0
98292025-04-30T06:43:05.865ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "f75cff7c-6e4d-445a-9394-81b5f7e0c569", "content-length": "169", "date": "Wed, 30 Apr 2025 06:43:05 GMT"} })
98302025-04-30T06:43:05.865ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/0/component/rot/caboose?firmware_slot=1
98312025-04-30T06:43:05.866ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
98322025-04-30T06:43:05.866ZTRACtest_snapshot_basic (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
98332025-04-30T06:43:05.866ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
98342025-04-30T06:43:05.866ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {} }
98352025-04-30T06:43:05.867ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
98362025-04-30T06:43:05.871ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "6dcbf66f-b4a5-4cb2-b412-02ce6c743a2d", "content-length": "169", "date": "Wed, 30 Apr 2025 06:43:05 GMT"} })
98372025-04-30T06:43:05.871ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/0/component/stage0/caboose?firmware_slot=0
98382025-04-30T06:43:05.883ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
98392025-04-30T06:43:05.894ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
98402025-04-30T06:43:05.894ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
98412025-04-30T06:43:05.894ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {} }
98422025-04-30T06:43:05.894ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
98432025-04-30T06:43:05.895ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "7cde419d-559c-43e0-9f01-bfaac09713d6", "content-length": "178", "date": "Wed, 30 Apr 2025 06:43:05 GMT"} })
98442025-04-30T06:43:05.895ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/0/component/stage0/caboose?firmware_slot=1
98452025-04-30T06:43:05.900ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "cfee1c73-d2d2-4d3e-874c-e3b0c6fd44f3", "content-length": "178", "date": "Wed, 30 Apr 2025 06:43:05 GMT"} })
98462025-04-30T06:43:05.900ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/0/component/rot/cmpa
98472025-04-30T06:43:05.900ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
98482025-04-30T06:43:05.901ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {} }
98492025-04-30T06:43:05.901ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
98502025-04-30T06:43:05.903ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "900d7457-fe4d-4e0a-9f8d-014780f2aef4", "content-length": "702", "date": "Wed, 30 Apr 2025 06:43:05 GMT"} })
98512025-04-30T06:43:05.903ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/0/component/rot/cfpa
98522025-04-30T06:43:05.905ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "c1b97585-19ee-4e54-b727-aad66245fe59", "content-length": "727", "date": "Wed, 30 Apr 2025 06:43:05 GMT"} })
98532025-04-30T06:43:05.905ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/0/component/rot/cfpa
98542025-04-30T06:43:05.907ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "02c2cf74-fbc8-43ff-a0b4-080fe2a27f51", "content-length": "729", "date": "Wed, 30 Apr 2025 06:43:05 GMT"} })
98552025-04-30T06:43:05.907ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/0/component/rot/cfpa
98562025-04-30T06:43:05.909ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "d35631dc-d697-4c92-aec1-f4c9d587b09b", "content-length": "728", "date": "Wed, 30 Apr 2025 06:43:05 GMT"} })
98572025-04-30T06:43:05.909ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/1
98582025-04-30T06:43:05.912ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "266aa101-427a-4875-879f-c2964fc15f10", "content-length": "723", "date": "Wed, 30 Apr 2025 06:43:05 GMT"} })
98592025-04-30T06:43:05.912ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/1/component/sp/caboose?firmware_slot=0
98602025-04-30T06:43:05.917ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "30b88882-1e73-43e7-bd46-7e72905a073f", "content-length": "109", "date": "Wed, 30 Apr 2025 06:43:05 GMT"} })
98612025-04-30T06:43:05.917ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/1/component/sp/caboose?firmware_slot=1
98622025-04-30T06:43:05.922ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "87b26ada-cf2c-49dc-95b1-11c18fe7118c", "content-length": "109", "date": "Wed, 30 Apr 2025 06:43:05 GMT"} })
98632025-04-30T06:43:05.922ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/1/component/rot/caboose?firmware_slot=0
98642025-04-30T06:43:05.935ZDEBGtest_snapshot_basic: attempt to fetch latest inventory collection result = Ok("found")
98652025-04-30T06:43:05.935ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "1ae37b67-f770-4d94-9a52-86142d81f06f", "content-length": "169", "date": "Wed, 30 Apr 2025 06:43:05 GMT"} })
98662025-04-30T06:43:05.936ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/1/component/rot/caboose?firmware_slot=1
98672025-04-30T06:43:05.942ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "64af0136-88f5-449d-ab61-749ff0144b28", "content-length": "169", "date": "Wed, 30 Apr 2025 06:43:05 GMT"} })
98682025-04-30T06:43:05.942ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/1/component/stage0/caboose?firmware_slot=0
98692025-04-30T06:43:05.947ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "ef55ff7c-ef2f-4977-8e20-b480088a0d78", "content-length": "178", "date": "Wed, 30 Apr 2025 06:43:05 GMT"} })
98702025-04-30T06:43:05.947ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/1/component/stage0/caboose?firmware_slot=1
98712025-04-30T06:43:05.951ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "b58e7750-2890-4ac8-b403-9c9dd6c916bc", "content-length": "178", "date": "Wed, 30 Apr 2025 06:43:05 GMT"} })
98722025-04-30T06:43:05.951ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/1/component/rot/cmpa
98732025-04-30T06:43:05.952ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "c90bfa29-728e-4a2f-81ad-e9a37d882085", "content-length": "702", "date": "Wed, 30 Apr 2025 06:43:05 GMT"} })
98742025-04-30T06:43:05.952ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/1/component/rot/cfpa
98752025-04-30T06:43:05.954ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "fb519edf-6fc4-403b-9901-4c37819c10cc", "content-length": "727", "date": "Wed, 30 Apr 2025 06:43:05 GMT"} })
98762025-04-30T06:43:05.954ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/1/component/rot/cfpa
98772025-04-30T06:43:05.955ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "14062f5a-d0b3-4ec6-b11e-93319e60b185", "content-length": "729", "date": "Wed, 30 Apr 2025 06:43:05 GMT"} })
98782025-04-30T06:43:05.955ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/1/component/rot/cfpa
98792025-04-30T06:43:05.957ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "ab87eb52-2afa-420b-b026-258406549844", "content-length": "728", "date": "Wed, 30 Apr 2025 06:43:05 GMT"} })
98802025-04-30T06:43:05.957ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
98812025-04-30T06:43:05.957ZTRACtest_snapshot_basic (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
98822025-04-30T06:43:05.957ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
98832025-04-30T06:43:05.957ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
98842025-04-30T06:43:05.958ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
98852025-04-30T06:43:05.963ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
98862025-04-30T06:43:05.972ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
98872025-04-30T06:43:05.973ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
98882025-04-30T06:43:05.973ZTRACtest_snapshot_basic (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
98892025-04-30T06:43:05.973ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
98902025-04-30T06:43:05.973ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
98912025-04-30T06:43:05.974ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
98922025-04-30T06:43:05.986ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
98932025-04-30T06:43:06.015ZTRACtest_snapshot_basic (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
98942025-04-30T06:43:06.015ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
98952025-04-30T06:43:06.017ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {} }
98962025-04-30T06:43:06.018ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
98972025-04-30T06:43:06.018ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
98982025-04-30T06:43:06.024ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
98992025-04-30T06:43:06.024ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
99002025-04-30T06:43:06.024ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
99012025-04-30T06:43:06.025ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
99022025-04-30T06:43:06.028ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
99032025-04-30T06:43:06.039ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
99042025-04-30T06:43:06.039ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
99052025-04-30T06:43:06.039ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {} }
99062025-04-30T06:43:06.042ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
99072025-04-30T06:43:06.042ZDEBGtest_snapshot_basic (results-sink): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
99082025-04-30T06:43:06.042ZDEBGtest_snapshot_basic (results-sink-cluster): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
99092025-04-30T06:43:06.043ZDEBGtest_snapshot_basic (clickhouse-client): unrolling 1 total samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 6ad06c6b-9a6e-4ed1-b3f4-5d2e9da3f8a7
99102025-04-30T06:43:06.043ZTRACtest_snapshot_basic (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 6ad06c6b-9a6e-4ed1-b3f4-5d2e9da3f8a7 n_columns = 4 n_rows = 2 sql = INSERT INTO oximeter.fields_string FORMAT Native
99112025-04-30T06:43:06.052ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
99122025-04-30T06:43:06.052ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {} }
99132025-04-30T06:43:06.053ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
99142025-04-30T06:43:06.150ZDEBGtest_snapshot_basic (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:41923
99152025-04-30T06:43:06.150ZDEBGtest_snapshot_basic (BackgroundTasks): client request SledAgent = http://[::1]:41923 background_task = inventory_collection body = None method = GET uri = http://[::1]:41923/inventory
99162025-04-30T06:43:06.151ZINFOtest_snapshot_basic (dropshot): accepted connection local_addr = [::1]:41923 remote_addr = [::1]:41272 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
99172025-04-30T06:43:06.151ZDEBGtest_snapshot_basic (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 6ad06c6b-9a6e-4ed1-b3f4-5d2e9da3f8a7 n_rows = 2 table_name = fields_string
99182025-04-30T06:43:06.151ZTRACtest_snapshot_basic (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 6ad06c6b-9a6e-4ed1-b3f4-5d2e9da3f8a7 n_columns = 4 n_rows = 1 sql = INSERT INTO oximeter.measurements_i64 FORMAT Native
99192025-04-30T06:43:06.152ZWARNtest_snapshot_basic (results-sink-cluster): failed to insert some results into metric DB collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 error = Failed to check out connection to database: No backends found for this service file = oximeter/collector/src/results_sink.rs:92
99202025-04-30T06:43:06.153ZTRACtest_snapshot_basic (dropshot): incoming request local_addr = [::1]:41923 method = GET remote_addr = [::1]:41272 req_id = e866aeac-32d3-4ed6-8d29-49b28fbf842a sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
99212025-04-30T06:43:06.154ZINFOtest_snapshot_basic (dropshot): request completed latency_us = 1117 local_addr = [::1]:41923 method = GET remote_addr = [::1]:41272 req_id = e866aeac-32d3-4ed6-8d29-49b28fbf842a response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
99222025-04-30T06:43:06.155ZDEBGtest_snapshot_basic (BackgroundTasks): client response SledAgent = http://[::1]:41923 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:41923/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "e866aeac-32d3-4ed6-8d29-49b28fbf842a", "content-length": "3141", "date": "Wed, 30 Apr 2025 06:43:05 GMT"} })
99232025-04-30T06:43:06.160ZDEBGtest_snapshot_basic (BackgroundTasks): begin collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
99242025-04-30T06:43:06.160ZDEBGtest_snapshot_basic (BackgroundTasks): end collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
99252025-04-30T06:43:06.160ZDEBGtest_snapshot_basic (BackgroundTasks): finished collection background_task = inventory_collection
99262025-04-30T06:43:06.162ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
99272025-04-30T06:43:06.164ZTRACtest_snapshot_basic (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
99282025-04-30T06:43:06.164ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
99292025-04-30T06:43:06.164ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
99302025-04-30T06:43:06.165ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
99312025-04-30T06:43:06.166ZDEBGtest_snapshot_basic (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 6ad06c6b-9a6e-4ed1-b3f4-5d2e9da3f8a7 n_rows = 1 table_name = measurements_i64
99322025-04-30T06:43:06.166ZTRACtest_snapshot_basic (results-sink): successfully inserted samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
99332025-04-30T06:43:06.169ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
99342025-04-30T06:43:06.175ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
99352025-04-30T06:43:06.175ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
99362025-04-30T06:43:06.175ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
99372025-04-30T06:43:06.175ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
99382025-04-30T06:43:06.434ZDEBGtest_snapshot_basic: attempt to fetch latest inventory collection result = Ok("found")
99392025-04-30T06:43:06.434ZWARNtest_snapshot_basic (datastore): Non-retryable transaction failure err = DatabaseError(UniqueViolation, "duplicate key value violates unique constraint \\"inv_caboose_pkey\\"") transaction = inventory_insert_collection
99402025-04-30T06:43:06.434ZWARNtest_snapshot_basic (BackgroundTasks): inventory collection failed background_task = inventory_collection error = failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint "inv_caboose_pkey"
99412025-04-30T06:43:06.434ZDEBGtest_snapshot_basic (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 899.879561ms iteration = 34 status = {"error":"failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint \\"inv_caboose_pkey\\""}
99422025-04-30T06:43:06.434ZDEBGtest_snapshot_basic (BackgroundTasks): activating background_task = inventory_collection iteration = 35 reason = Signaled
99432025-04-30T06:43:06.434ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
99442025-04-30T06:43:06.434ZTRACtest_snapshot_basic (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
99452025-04-30T06:43:06.434ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
99462025-04-30T06:43:06.435ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
99472025-04-30T06:43:06.435ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
99482025-04-30T06:43:06.443ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
99492025-04-30T06:43:06.446ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
99502025-04-30T06:43:06.446ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
99512025-04-30T06:43:06.446ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
99522025-04-30T06:43:06.447ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
99532025-04-30T06:43:06.450ZDEBGtest_snapshot_basic (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = [(d1ad44e5-18dd-4193-a70e-bf9042f0e0e1, 0)]
99542025-04-30T06:43:06.450ZTRACtest_snapshot_basic (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
99552025-04-30T06:43:06.451ZTRACtest_snapshot_basic (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: 85 24 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 .$..........._mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:53465 req_id = b3bc8959-8f2d-449f-b24c-bdaf29b5f92b
99562025-04-30T06:43:06.451ZDEBGtest_snapshot_basic (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 34084,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:53465 req_id = b3bc8959-8f2d-449f-b24c-bdaf29b5f92b
99572025-04-30T06:43:06.451ZDEBGtest_snapshot_basic (store): query key key = _mgs._tcp
99582025-04-30T06:43:06.451ZDEBGtest_snapshot_basic (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
99592025-04-30T06:43:06.451ZDEBGtest_snapshot_basic (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:53465 query = LowerQuery { name: LowerName(Name("_mgs._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 41502, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = b3bc8959-8f2d-449f-b24c-bdaf29b5f92b
99602025-04-30T06:43:06.452ZDEBGtest_snapshot_basic (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 41502, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 4251, tv_nsec: 250989156 } })
99612025-04-30T06:43:06.452ZTRACtest_snapshot_basic (DnsResolver): lookup_service_targets: looking up SRV target
99622025-04-30T06:43:06.452ZTRACtest_snapshot_basic (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: 29 4f 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e )O..........-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:64999 req_id = d648f562-0c8a-4816-8997-8fae4abbc33a
99632025-04-30T06:43:06.452ZDEBGtest_snapshot_basic (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 10575,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:64999 req_id = d648f562-0c8a-4816-8997-8fae4abbc33a
99642025-04-30T06:43:06.452ZDEBGtest_snapshot_basic (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
99652025-04-30T06:43:06.452ZDEBGtest_snapshot_basic (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:64999 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = d648f562-0c8a-4816-8997-8fae4abbc33a
99662025-04-30T06:43:06.550ZTRACtest_snapshot_basic (DnsResolver): lookup_all_socket_v6 srv dns_name = _clickhouse-admin-keeper._tcp.control-plane.oxide.internal
99672025-04-30T06:43:06.553ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
99682025-04-30T06:43:06.553ZTRACtest_snapshot_basic (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
99692025-04-30T06:43:06.553ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
99702025-04-30T06:43:06.553ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {} }
99712025-04-30T06:43:06.554ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
99722025-04-30T06:43:06.556ZTRACtest_snapshot_basic (dns): buffer buffer = Length: 87 (0x57) bytes\n0000: 9a 52 01 00 00 01 00 00 00 00 00 01 18 5f 63 6c .R..........._cl\n0010: 69 63 6b 68 6f 75 73 65 2d 61 64 6d 69 6e 2d 6b ickhouse-admin-k\n0020: 65 65 70 65 72 04 5f 74 63 70 0d 63 6f 6e 74 72 eeper._tcp.contr\n0030: 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 69 ol-plane.oxide.i\n0040: 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 04 nternal..!....).\n0050: d0 00 00 00 00 00 00 ....... kind = dns peer_addr = [::1]:53770 req_id = ab4104f7-03f9-461c-a33f-8f0b54d6fbc5
99732025-04-30T06:43:06.556ZDEBGtest_snapshot_basic (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 39506,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 24,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 97,\n 100,\n 109,\n 105,\n 110,\n 45,\n 107,\n 101,\n 101,\n 112,\n 101,\n 114,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:53770 req_id = ab4104f7-03f9-461c-a33f-8f0b54d6fbc5
99742025-04-30T06:43:06.556ZDEBGtest_snapshot_basic (store): query key key = _clickhouse-admin-keeper._tcp
99752025-04-30T06:43:06.556ZERROtest_snapshot_basic (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 39506, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } NXDOMAIN: no records found for name: "_clickhouse-admin-keeper._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:53770 req_id = ab4104f7-03f9-461c-a33f-8f0b54d6fbc5
99762025-04-30T06:43:06.560ZDEBGtest_snapshot_basic (BackgroundTasks): begin collection background_task = inventory_collection
99772025-04-30T06:43:06.560ZDEBGtest_snapshot_basic (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:41502
99782025-04-30T06:43:06.560ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/ignition
99792025-04-30T06:43:06.561ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
99802025-04-30T06:43:06.570ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
99812025-04-30T06:43:06.570ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
99822025-04-30T06:43:06.570ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {} }
99832025-04-30T06:43:06.571ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
99842025-04-30T06:43:06.588ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
99852025-04-30T06:43:06.588ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {} }
99862025-04-30T06:43:06.589ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
99872025-04-30T06:43:06.597ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "ea327e49-d2f1-49bf-b24b-1e223146325b", "content-length": "835", "date": "Wed, 30 Apr 2025 06:43:05 GMT"} })
99882025-04-30T06:43:06.597ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/0
99892025-04-30T06:43:06.689ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "fd4df609-e7bf-4b5f-b3bf-3daed4e7cbf9", "content-length": "734", "date": "Wed, 30 Apr 2025 06:43:05 GMT"} })
99902025-04-30T06:43:06.689ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/0/component/sp/caboose?firmware_slot=0
99912025-04-30T06:43:06.704ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "634539f1-26c5-4c5d-9866-53ed86a6a065", "content-length": "111", "date": "Wed, 30 Apr 2025 06:43:05 GMT"} })
99922025-04-30T06:43:06.704ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/0/component/sp/caboose?firmware_slot=1
99932025-04-30T06:43:06.712ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "d9be7360-b630-4db4-abed-4c132944d802", "content-length": "111", "date": "Wed, 30 Apr 2025 06:43:05 GMT"} })
99942025-04-30T06:43:06.712ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/0/component/rot/caboose?firmware_slot=0
99952025-04-30T06:43:06.809ZDEBGtest_snapshot_basic: attempt to fetch latest inventory collection result = Ok("found")
99962025-04-30T06:43:06.810ZDEBGtest_snapshot_basic (collection-task): sent timer-based collection request to the collection task collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
99972025-04-30T06:43:06.812ZDEBGtest_snapshot_basic (collection-task): collecting from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
99982025-04-30T06:43:06.815ZTRACtest_snapshot_basic (collection-task): sent collection request to producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
99992025-04-30T06:43:06.816ZDEBGtest_snapshot_basic (collection-task): collected results from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 n_results = 1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
100002025-04-30T06:43:06.816ZTRACtest_snapshot_basic (collection-task): forwarded results to main collection loop collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
100012025-04-30T06:43:06.816ZTRACtest_snapshot_basic (collection-task): top of inner collection loop, waiting for next request collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
100022025-04-30T06:43:06.819ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "ff810dc2-aea7-4f70-b1c5-1f335e75faef", "content-length": "170", "date": "Wed, 30 Apr 2025 06:43:06 GMT"} })
100032025-04-30T06:43:06.820ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/0/component/rot/caboose?firmware_slot=1
100042025-04-30T06:43:06.824ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "da9a44fb-aaca-44a9-bcaa-a8562fefec67", "content-length": "170", "date": "Wed, 30 Apr 2025 06:43:06 GMT"} })
100052025-04-30T06:43:06.824ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/0/component/stage0/caboose?firmware_slot=0
100062025-04-30T06:43:06.829ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "6c5df8dc-0039-454d-9cad-059b1a889bcc", "content-length": "179", "date": "Wed, 30 Apr 2025 06:43:06 GMT"} })
100072025-04-30T06:43:06.830ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/0/component/stage0/caboose?firmware_slot=1
100082025-04-30T06:43:06.834ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "c5b80a7e-a6eb-469c-80dc-62303b0b6cc7", "content-length": "179", "date": "Wed, 30 Apr 2025 06:43:06 GMT"} })
100092025-04-30T06:43:06.834ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/0/component/rot/cmpa
100102025-04-30T06:43:06.835ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "8baa0a65-98ca-4aa6-b7eb-2ed4db326806", "content-length": "702", "date": "Wed, 30 Apr 2025 06:43:06 GMT"} })
100112025-04-30T06:43:06.836ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/0/component/rot/cfpa
100122025-04-30T06:43:06.839ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "f650ce63-e452-42ff-8723-29447a28dbec", "content-length": "727", "date": "Wed, 30 Apr 2025 06:43:06 GMT"} })
100132025-04-30T06:43:06.840ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/0/component/rot/cfpa
100142025-04-30T06:43:06.842ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "29f28d7c-2df5-4797-a9f7-d6442a668288", "content-length": "729", "date": "Wed, 30 Apr 2025 06:43:06 GMT"} })
100152025-04-30T06:43:06.844ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/0/component/rot/cfpa
100162025-04-30T06:43:06.845ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "551ce9d6-1f52-471e-b866-2f4e6205f4a2", "content-length": "728", "date": "Wed, 30 Apr 2025 06:43:06 GMT"} })
100172025-04-30T06:43:06.846ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/1
100182025-04-30T06:43:06.848ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "18896027-33eb-469c-ab53-6d88e48b83f7", "content-length": "734", "date": "Wed, 30 Apr 2025 06:43:06 GMT"} })
100192025-04-30T06:43:06.848ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/1/component/sp/caboose?firmware_slot=0
100202025-04-30T06:43:06.850ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "60671fbe-7bda-40fa-a50f-5740a83ea8bf", "content-length": "111", "date": "Wed, 30 Apr 2025 06:43:06 GMT"} })
100212025-04-30T06:43:06.851ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/1/component/sp/caboose?firmware_slot=1
100222025-04-30T06:43:06.855ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "28b10547-7ead-4566-a14c-55f4d87cdfd1", "content-length": "111", "date": "Wed, 30 Apr 2025 06:43:06 GMT"} })
100232025-04-30T06:43:06.855ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/1/component/rot/caboose?firmware_slot=0
100242025-04-30T06:43:06.857ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "bb97797d-0340-4ba2-8bcf-646253c94ec5", "content-length": "170", "date": "Wed, 30 Apr 2025 06:43:06 GMT"} })
100252025-04-30T06:43:06.857ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/1/component/rot/caboose?firmware_slot=1
100262025-04-30T06:43:06.859ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "5083cc5a-0e78-4418-a1ad-475472048524", "content-length": "170", "date": "Wed, 30 Apr 2025 06:43:06 GMT"} })
100272025-04-30T06:43:06.859ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/1/component/stage0/caboose?firmware_slot=0
100282025-04-30T06:43:06.860ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
100292025-04-30T06:43:06.860ZTRACtest_snapshot_basic (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
100302025-04-30T06:43:06.860ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
100312025-04-30T06:43:06.860ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {} }
100322025-04-30T06:43:06.861ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
100332025-04-30T06:43:06.868ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
100342025-04-30T06:43:06.875ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
100352025-04-30T06:43:06.894ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
100362025-04-30T06:43:06.894ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {} }
100372025-04-30T06:43:06.901ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
100382025-04-30T06:43:06.901ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "8caad54d-9ab6-4454-827d-68df5c7ee5ca", "content-length": "179", "date": "Wed, 30 Apr 2025 06:43:06 GMT"} })
100392025-04-30T06:43:06.901ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/1/component/stage0/caboose?firmware_slot=1
100402025-04-30T06:43:06.908ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "6921a51e-1298-4c20-aaca-0b6292a1868a", "content-length": "179", "date": "Wed, 30 Apr 2025 06:43:06 GMT"} })
100412025-04-30T06:43:06.908ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/1/component/rot/cmpa
100422025-04-30T06:43:06.908ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
100432025-04-30T06:43:06.908ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {} }
100442025-04-30T06:43:06.909ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
100452025-04-30T06:43:06.936ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "f4512db1-94ce-4204-a6d0-0ffa8104f8cb", "content-length": "702", "date": "Wed, 30 Apr 2025 06:43:06 GMT"} })
100462025-04-30T06:43:06.937ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/1/component/rot/cfpa
100472025-04-30T06:43:06.941ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "50a11c8e-7afd-4ad2-b4e0-1d02d71d2c3e", "content-length": "727", "date": "Wed, 30 Apr 2025 06:43:06 GMT"} })
100482025-04-30T06:43:06.942ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/1/component/rot/cfpa
100492025-04-30T06:43:06.943ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "ad44f29f-d5af-42d6-9647-94a4396b022f", "content-length": "729", "date": "Wed, 30 Apr 2025 06:43:06 GMT"} })
100502025-04-30T06:43:06.943ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/1/component/rot/cfpa
100512025-04-30T06:43:06.944ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "30daa1e0-4650-41c8-8dad-31723c7a4725", "content-length": "728", "date": "Wed, 30 Apr 2025 06:43:06 GMT"} })
100522025-04-30T06:43:06.944ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/0
100532025-04-30T06:43:06.946ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "fa55c4c4-1b3a-455a-b2f1-84f5c254751d", "content-length": "723", "date": "Wed, 30 Apr 2025 06:43:06 GMT"} })
100542025-04-30T06:43:06.946ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/0/component/sp/caboose?firmware_slot=0
100552025-04-30T06:43:06.949ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "a8780b23-7893-4c80-a390-bad421cbc458", "content-length": "109", "date": "Wed, 30 Apr 2025 06:43:06 GMT"} })
100562025-04-30T06:43:06.949ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/0/component/sp/caboose?firmware_slot=1
100572025-04-30T06:43:06.952ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "d4e337f4-c02a-4778-bd26-4e1f5d766e2e", "content-length": "109", "date": "Wed, 30 Apr 2025 06:43:06 GMT"} })
100582025-04-30T06:43:06.952ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/0/component/rot/caboose?firmware_slot=0
100592025-04-30T06:43:06.962ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "60c78343-75d9-4c9f-be20-2a53024afb4e", "content-length": "169", "date": "Wed, 30 Apr 2025 06:43:06 GMT"} })
100602025-04-30T06:43:06.962ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/0/component/rot/caboose?firmware_slot=1
100612025-04-30T06:43:06.978ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "173f6468-685e-4711-93ab-85a4fc59451a", "content-length": "169", "date": "Wed, 30 Apr 2025 06:43:06 GMT"} })
100622025-04-30T06:43:06.978ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/0/component/stage0/caboose?firmware_slot=0
100632025-04-30T06:43:06.979ZDEBGtest_snapshot_basic: attempt to fetch latest inventory collection result = Ok("found")
100642025-04-30T06:43:07.046ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
100652025-04-30T06:43:07.046ZTRACtest_snapshot_basic (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
100662025-04-30T06:43:07.047ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
100672025-04-30T06:43:07.047ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {} }
100682025-04-30T06:43:07.048ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
100692025-04-30T06:43:07.048ZDEBGtest_snapshot_basic (results-sink-cluster): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
100702025-04-30T06:43:07.048ZDEBGtest_snapshot_basic (results-sink): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
100712025-04-30T06:43:07.048ZDEBGtest_snapshot_basic (clickhouse-client): unrolling 1 total samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 6ad06c6b-9a6e-4ed1-b3f4-5d2e9da3f8a7
100722025-04-30T06:43:07.050ZTRACtest_snapshot_basic (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 6ad06c6b-9a6e-4ed1-b3f4-5d2e9da3f8a7 n_columns = 4 n_rows = 2 sql = INSERT INTO oximeter.fields_string FORMAT Native
100732025-04-30T06:43:07.051ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
100742025-04-30T06:43:07.057ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
100752025-04-30T06:43:07.057ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
100762025-04-30T06:43:07.057ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {} }
100772025-04-30T06:43:07.058ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
100782025-04-30T06:43:07.059ZDEBGtest_snapshot_basic (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 6ad06c6b-9a6e-4ed1-b3f4-5d2e9da3f8a7 n_rows = 2 table_name = fields_string
100792025-04-30T06:43:07.059ZTRACtest_snapshot_basic (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 6ad06c6b-9a6e-4ed1-b3f4-5d2e9da3f8a7 n_columns = 4 n_rows = 1 sql = INSERT INTO oximeter.measurements_i64 FORMAT Native
100802025-04-30T06:43:07.092ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
100812025-04-30T06:43:07.092ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {} }
100822025-04-30T06:43:07.094ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
100832025-04-30T06:43:07.097ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "4b97b92d-9e5c-4adf-a82a-4ae6609b4ece", "content-length": "178", "date": "Wed, 30 Apr 2025 06:43:06 GMT"} })
100842025-04-30T06:43:07.097ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/0/component/stage0/caboose?firmware_slot=1
100852025-04-30T06:43:07.097ZDEBGtest_snapshot_basic (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 6ad06c6b-9a6e-4ed1-b3f4-5d2e9da3f8a7 n_rows = 1 table_name = measurements_i64
100862025-04-30T06:43:07.099ZTRACtest_snapshot_basic (results-sink): successfully inserted samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
100872025-04-30T06:43:07.131ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "ebf1ec3c-bd7e-4677-8d40-51bc97972973", "content-length": "178", "date": "Wed, 30 Apr 2025 06:43:06 GMT"} })
100882025-04-30T06:43:07.132ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/0/component/rot/cmpa
100892025-04-30T06:43:07.145ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "598078e0-0e27-49b8-bfac-db31c7e387e2", "content-length": "702", "date": "Wed, 30 Apr 2025 06:43:06 GMT"} })
100902025-04-30T06:43:07.145ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/0/component/rot/cfpa
100912025-04-30T06:43:07.152ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "b67caa35-7e31-43ab-953c-9cefca1a9945", "content-length": "727", "date": "Wed, 30 Apr 2025 06:43:06 GMT"} })
100922025-04-30T06:43:07.152ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/0/component/rot/cfpa
100932025-04-30T06:43:07.152ZWARNtest_snapshot_basic (results-sink-cluster): failed to insert some results into metric DB collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 error = Failed to check out connection to database: No backends found for this service file = oximeter/collector/src/results_sink.rs:92
100942025-04-30T06:43:07.158ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "c68539a5-ec39-4921-beb0-dd088e464717", "content-length": "729", "date": "Wed, 30 Apr 2025 06:43:06 GMT"} })
100952025-04-30T06:43:07.159ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/0/component/rot/cfpa
100962025-04-30T06:43:07.166ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "da5bc190-1738-4e44-af97-41d40253304c", "content-length": "728", "date": "Wed, 30 Apr 2025 06:43:06 GMT"} })
100972025-04-30T06:43:07.172ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/1
100982025-04-30T06:43:07.178ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "214eb4bc-fc9e-4bb3-876b-0b22d1a47b47", "content-length": "723", "date": "Wed, 30 Apr 2025 06:43:06 GMT"} })
100992025-04-30T06:43:07.179ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/1/component/sp/caboose?firmware_slot=0
101002025-04-30T06:43:07.184ZDEBGtest_snapshot_basic: attempt to fetch latest inventory collection result = Ok("found")
101012025-04-30T06:43:07.212ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "1d9afd12-a870-4937-afab-7c3f98e78223", "content-length": "109", "date": "Wed, 30 Apr 2025 06:43:06 GMT"} })
101022025-04-30T06:43:07.212ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/1/component/sp/caboose?firmware_slot=1
101032025-04-30T06:43:07.214ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "ab6f048a-17c3-4f1a-9698-645ca316d94e", "content-length": "109", "date": "Wed, 30 Apr 2025 06:43:06 GMT"} })
101042025-04-30T06:43:07.215ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/1/component/rot/caboose?firmware_slot=0
101052025-04-30T06:43:07.217ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "cb8fda1b-c20b-4576-9b5b-e9003640d496", "content-length": "169", "date": "Wed, 30 Apr 2025 06:43:06 GMT"} })
101062025-04-30T06:43:07.221ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/1/component/rot/caboose?firmware_slot=1
101072025-04-30T06:43:07.223ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "062bb25d-b921-48ba-9272-433348b2f0b3", "content-length": "169", "date": "Wed, 30 Apr 2025 06:43:06 GMT"} })
101082025-04-30T06:43:07.223ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/1/component/stage0/caboose?firmware_slot=0
101092025-04-30T06:43:07.226ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "0d872065-ec4a-4791-9ec3-84e575fae655", "content-length": "178", "date": "Wed, 30 Apr 2025 06:43:06 GMT"} })
101102025-04-30T06:43:07.226ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/1/component/stage0/caboose?firmware_slot=1
101112025-04-30T06:43:07.253ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
101122025-04-30T06:43:07.254ZTRACtest_snapshot_basic (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
101132025-04-30T06:43:07.258ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
101142025-04-30T06:43:07.265ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {} }
101152025-04-30T06:43:07.266ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
101162025-04-30T06:43:07.268ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "f0d35091-c97c-49d1-9fec-a2adcec10c98", "content-length": "178", "date": "Wed, 30 Apr 2025 06:43:06 GMT"} })
101172025-04-30T06:43:07.269ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/1/component/rot/cmpa
101182025-04-30T06:43:07.269ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
101192025-04-30T06:43:07.278ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
101202025-04-30T06:43:07.278ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
101212025-04-30T06:43:07.278ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {} }
101222025-04-30T06:43:07.278ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
101232025-04-30T06:43:07.279ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "32a7b216-c183-4dbe-8052-59b2fbe9c415", "content-length": "702", "date": "Wed, 30 Apr 2025 06:43:06 GMT"} })
101242025-04-30T06:43:07.279ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/1/component/rot/cfpa
101252025-04-30T06:43:07.282ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "a9ea6a1a-6d73-4781-bc13-362d96db9934", "content-length": "727", "date": "Wed, 30 Apr 2025 06:43:06 GMT"} })
101262025-04-30T06:43:07.283ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/1/component/rot/cfpa
101272025-04-30T06:43:07.286ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "77aca584-d497-4289-9a57-097f0e23d2dc", "content-length": "729", "date": "Wed, 30 Apr 2025 06:43:06 GMT"} })
101282025-04-30T06:43:07.286ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/1/component/rot/cfpa
101292025-04-30T06:43:07.287ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "72e80e85-2bd7-417c-9b2f-f588165ed02e", "content-length": "728", "date": "Wed, 30 Apr 2025 06:43:06 GMT"} })
101302025-04-30T06:43:07.287ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
101312025-04-30T06:43:07.287ZTRACtest_snapshot_basic (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
101322025-04-30T06:43:07.287ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
101332025-04-30T06:43:07.287ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
101342025-04-30T06:43:07.288ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
101352025-04-30T06:43:07.303ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
101362025-04-30T06:43:07.303ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {} }
101372025-04-30T06:43:07.304ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
101382025-04-30T06:43:07.305ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
101392025-04-30T06:43:07.310ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
101402025-04-30T06:43:07.310ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
101412025-04-30T06:43:07.310ZTRACtest_snapshot_basic (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
101422025-04-30T06:43:07.310ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
101432025-04-30T06:43:07.310ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
101442025-04-30T06:43:07.312ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
101452025-04-30T06:43:07.318ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
101462025-04-30T06:43:07.328ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
101472025-04-30T06:43:07.329ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
101482025-04-30T06:43:07.329ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
101492025-04-30T06:43:07.329ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
101502025-04-30T06:43:07.421ZDEBGtest_snapshot_basic (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:41923
101512025-04-30T06:43:07.421ZDEBGtest_snapshot_basic (BackgroundTasks): client request SledAgent = http://[::1]:41923 background_task = inventory_collection body = None method = GET uri = http://[::1]:41923/inventory
101522025-04-30T06:43:07.422ZINFOtest_snapshot_basic (dropshot): accepted connection local_addr = [::1]:41923 remote_addr = [::1]:52217 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
101532025-04-30T06:43:07.424ZTRACtest_snapshot_basic (dropshot): incoming request local_addr = [::1]:41923 method = GET remote_addr = [::1]:52217 req_id = 968706d7-fdfd-40bf-a400-f227515da789 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
101542025-04-30T06:43:07.426ZINFOtest_snapshot_basic (dropshot): request completed latency_us = 1652 local_addr = [::1]:41923 method = GET remote_addr = [::1]:52217 req_id = 968706d7-fdfd-40bf-a400-f227515da789 response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
101552025-04-30T06:43:07.427ZDEBGtest_snapshot_basic (BackgroundTasks): client response SledAgent = http://[::1]:41923 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:41923/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "968706d7-fdfd-40bf-a400-f227515da789", "content-length": "3141", "date": "Wed, 30 Apr 2025 06:43:06 GMT"} })
101562025-04-30T06:43:07.429ZDEBGtest_snapshot_basic (BackgroundTasks): begin collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
101572025-04-30T06:43:07.429ZDEBGtest_snapshot_basic (BackgroundTasks): end collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
101582025-04-30T06:43:07.429ZDEBGtest_snapshot_basic (BackgroundTasks): finished collection background_task = inventory_collection
101592025-04-30T06:43:07.431ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
101602025-04-30T06:43:07.431ZTRACtest_snapshot_basic (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
101612025-04-30T06:43:07.431ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
101622025-04-30T06:43:07.431ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
101632025-04-30T06:43:07.432ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
101642025-04-30T06:43:07.434ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
101652025-04-30T06:43:07.438ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
101662025-04-30T06:43:07.438ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
101672025-04-30T06:43:07.438ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
101682025-04-30T06:43:07.439ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
101692025-04-30T06:43:07.472ZDEBGtest_snapshot_basic: attempt to fetch latest inventory collection result = Ok("found")
101702025-04-30T06:43:07.493ZWARNtest_snapshot_basic (datastore): Non-retryable transaction failure err = DatabaseError(UniqueViolation, "duplicate key value violates unique constraint \\"inv_caboose_pkey\\"") transaction = inventory_insert_collection
101712025-04-30T06:43:07.493ZWARNtest_snapshot_basic (BackgroundTasks): inventory collection failed background_task = inventory_collection error = failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint "inv_caboose_pkey"
101722025-04-30T06:43:07.493ZDEBGtest_snapshot_basic (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 1.058558618s iteration = 35 status = {"error":"failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint \\"inv_caboose_pkey\\""}
101732025-04-30T06:43:07.493ZDEBGtest_snapshot_basic (BackgroundTasks): activating background_task = inventory_collection iteration = 36 reason = Signaled
101742025-04-30T06:43:07.493ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
101752025-04-30T06:43:07.493ZTRACtest_snapshot_basic (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
101762025-04-30T06:43:07.493ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
101772025-04-30T06:43:07.493ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
101782025-04-30T06:43:07.494ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
101792025-04-30T06:43:07.500ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
101802025-04-30T06:43:07.503ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
101812025-04-30T06:43:07.503ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
101822025-04-30T06:43:07.503ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
101832025-04-30T06:43:07.504ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
101842025-04-30T06:43:07.508ZDEBGtest_snapshot_basic (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = [(d1ad44e5-18dd-4193-a70e-bf9042f0e0e1, 0)]
101852025-04-30T06:43:07.508ZTRACtest_snapshot_basic (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
101862025-04-30T06:43:07.512ZTRACtest_snapshot_basic (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: 57 54 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 WT..........._mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:49456 req_id = 1596301f-49e8-4804-9de9-1c21e310364c
101872025-04-30T06:43:07.512ZDEBGtest_snapshot_basic (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 22356,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:49456 req_id = 1596301f-49e8-4804-9de9-1c21e310364c
101882025-04-30T06:43:07.512ZDEBGtest_snapshot_basic (store): query key key = _mgs._tcp
101892025-04-30T06:43:07.512ZDEBGtest_snapshot_basic (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
101902025-04-30T06:43:07.512ZDEBGtest_snapshot_basic (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:49456 query = LowerQuery { name: LowerName(Name("_mgs._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 41502, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = 1596301f-49e8-4804-9de9-1c21e310364c
101912025-04-30T06:43:07.515ZDEBGtest_snapshot_basic (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 41502, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 4252, tv_nsec: 314121107 } })
101922025-04-30T06:43:07.515ZTRACtest_snapshot_basic (DnsResolver): lookup_service_targets: looking up SRV target
101932025-04-30T06:43:07.521ZTRACtest_snapshot_basic (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: d2 ce 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e ............-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:57782 req_id = e888bafa-0f87-43c4-8e2e-3a810aceaf46
101942025-04-30T06:43:07.521ZDEBGtest_snapshot_basic (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 53966,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:57782 req_id = e888bafa-0f87-43c4-8e2e-3a810aceaf46
101952025-04-30T06:43:07.521ZDEBGtest_snapshot_basic (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
101962025-04-30T06:43:07.521ZDEBGtest_snapshot_basic (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:57782 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = e888bafa-0f87-43c4-8e2e-3a810aceaf46
101972025-04-30T06:43:07.594ZTRACtest_snapshot_basic (DnsResolver): lookup_all_socket_v6 srv dns_name = _clickhouse-admin-keeper._tcp.control-plane.oxide.internal
101982025-04-30T06:43:07.640ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
101992025-04-30T06:43:07.640ZTRACtest_snapshot_basic (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
102002025-04-30T06:43:07.640ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
102012025-04-30T06:43:07.640ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {} }
102022025-04-30T06:43:07.642ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
102032025-04-30T06:43:07.646ZTRACtest_snapshot_basic (dns): buffer buffer = Length: 87 (0x57) bytes\n0000: be 6b 01 00 00 01 00 00 00 00 00 01 18 5f 63 6c .k..........._cl\n0010: 69 63 6b 68 6f 75 73 65 2d 61 64 6d 69 6e 2d 6b ickhouse-admin-k\n0020: 65 65 70 65 72 04 5f 74 63 70 0d 63 6f 6e 74 72 eeper._tcp.contr\n0030: 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 69 ol-plane.oxide.i\n0040: 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 04 nternal..!....).\n0050: d0 00 00 00 00 00 00 ....... kind = dns peer_addr = [::1]:53732 req_id = 194e56c8-ac56-4ac3-bb85-e699472ed226
102042025-04-30T06:43:07.646ZDEBGtest_snapshot_basic (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 48747,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 24,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 97,\n 100,\n 109,\n 105,\n 110,\n 45,\n 107,\n 101,\n 101,\n 112,\n 101,\n 114,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:53732 req_id = 194e56c8-ac56-4ac3-bb85-e699472ed226
102052025-04-30T06:43:07.646ZDEBGtest_snapshot_basic (store): query key key = _clickhouse-admin-keeper._tcp
102062025-04-30T06:43:07.647ZERROtest_snapshot_basic (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 48747, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } NXDOMAIN: no records found for name: "_clickhouse-admin-keeper._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:53732 req_id = 194e56c8-ac56-4ac3-bb85-e699472ed226
102072025-04-30T06:43:07.648ZDEBGtest_snapshot_basic (BackgroundTasks): begin collection background_task = inventory_collection
102082025-04-30T06:43:07.648ZDEBGtest_snapshot_basic (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:41502
102092025-04-30T06:43:07.648ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/ignition
102102025-04-30T06:43:07.663ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
102112025-04-30T06:43:07.670ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
102122025-04-30T06:43:07.670ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
102132025-04-30T06:43:07.670ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {} }
102142025-04-30T06:43:07.670ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
102152025-04-30T06:43:07.688ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
102162025-04-30T06:43:07.688ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {} }
102172025-04-30T06:43:07.689ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
102182025-04-30T06:43:07.696ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "c702ab42-fc37-4b66-9eb6-40d5cce14af8", "content-length": "835", "date": "Wed, 30 Apr 2025 06:43:06 GMT"} })
102192025-04-30T06:43:07.696ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/0
102202025-04-30T06:43:07.713ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "7fc948d2-36c4-47a7-9e49-5975b7233344", "content-length": "734", "date": "Wed, 30 Apr 2025 06:43:06 GMT"} })
102212025-04-30T06:43:07.713ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/0/component/sp/caboose?firmware_slot=0
102222025-04-30T06:43:07.717ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "a315caf5-3617-44e6-9bd6-c292c0acdf28", "content-length": "111", "date": "Wed, 30 Apr 2025 06:43:06 GMT"} })
102232025-04-30T06:43:07.717ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/0/component/sp/caboose?firmware_slot=1
102242025-04-30T06:43:07.721ZINFOtest_snapshot_basic (oximeter-agent): refreshing list of producers from Nexus collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 file = oximeter/collector/src/agent.rs:562
102252025-04-30T06:43:07.721ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "d7972a9a-7937-4c51-b668-6afbf43a1366", "content-length": "111", "date": "Wed, 30 Apr 2025 06:43:06 GMT"} })
102262025-04-30T06:43:07.722ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/0/component/rot/caboose?firmware_slot=0
102272025-04-30T06:43:07.722ZDEBGtest_snapshot_basic (oximeter): client request body = None method = GET uri = http://127.0.0.1:61468/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
102282025-04-30T06:43:07.723ZTRACtest_snapshot_basic (dropshot_internal): incoming request local_addr = 127.0.0.1:61468 method = GET remote_addr = 127.0.0.1:64332 req_id = b05cddad-7b23-4bbc-aaa8-3bd71604a38c uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
102292025-04-30T06:43:07.724ZTRACtest_snapshot_basic (dropshot_internal): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:61468 method = GET remote_addr = 127.0.0.1:64332 req_id = b05cddad-7b23-4bbc-aaa8-3bd71604a38c resource = Database uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
102302025-04-30T06:43:07.724ZDEBGtest_snapshot_basic (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:61468 method = GET remote_addr = 127.0.0.1:64332 req_id = b05cddad-7b23-4bbc-aaa8-3bd71604a38c roles = RoleSet { roles: {} } uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
102312025-04-30T06:43:07.726ZDEBGtest_snapshot_basic (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:61468 method = GET remote_addr = 127.0.0.1:64332 req_id = b05cddad-7b23-4bbc-aaa8-3bd71604a38c resource = Database result = Ok(()) uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
102322025-04-30T06:43:07.740ZINFOtest_snapshot_basic (dropshot_internal): request completed latency_us = 16520 local_addr = 127.0.0.1:61468 method = GET remote_addr = 127.0.0.1:64332 req_id = b05cddad-7b23-4bbc-aaa8-3bd71604a38c response_code = 200 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
102332025-04-30T06:43:07.740ZDEBGtest_snapshot_basic (oximeter): client response result = Ok(Response { url: "http://127.0.0.1:61468/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending", status: 200, headers: {"content-type": "application/json", "x-request-id": "b05cddad-7b23-4bbc-aaa8-3bd71604a38c", "content-length": "405", "date": "Wed, 30 Apr 2025 06:43:06 GMT"} })
102342025-04-30T06:43:07.740ZDEBGtest_snapshot_basic (oximeter): client request body = None method = GET uri = http://127.0.0.1:61468/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
102352025-04-30T06:43:07.741ZTRACtest_snapshot_basic (dropshot_internal): incoming request local_addr = 127.0.0.1:61468 method = GET remote_addr = 127.0.0.1:64332 req_id = 6aacf17c-aa94-4935-975e-0596cd181bae uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
102362025-04-30T06:43:07.742ZTRACtest_snapshot_basic (dropshot_internal): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:61468 method = GET remote_addr = 127.0.0.1:64332 req_id = 6aacf17c-aa94-4935-975e-0596cd181bae resource = Database uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
102372025-04-30T06:43:07.742ZDEBGtest_snapshot_basic (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:61468 method = GET remote_addr = 127.0.0.1:64332 req_id = 6aacf17c-aa94-4935-975e-0596cd181bae roles = RoleSet { roles: {} } uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
102382025-04-30T06:43:07.743ZDEBGtest_snapshot_basic (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:61468 method = GET remote_addr = 127.0.0.1:64332 req_id = 6aacf17c-aa94-4935-975e-0596cd181bae resource = Database result = Ok(()) uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
102392025-04-30T06:43:07.745ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "53fcc15a-1f6f-4ffc-9732-544bc6cdcdb4", "content-length": "170", "date": "Wed, 30 Apr 2025 06:43:06 GMT"} })
102402025-04-30T06:43:07.745ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/0/component/rot/caboose?firmware_slot=1
102412025-04-30T06:43:07.747ZINFOtest_snapshot_basic (dropshot_internal): request completed latency_us = 5721 local_addr = 127.0.0.1:61468 method = GET remote_addr = 127.0.0.1:64332 req_id = 6aacf17c-aa94-4935-975e-0596cd181bae response_code = 200 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
102422025-04-30T06:43:07.748ZDEBGtest_snapshot_basic (oximeter): client response result = Ok(Response { url: "http://127.0.0.1:61468/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D", status: 200, headers: {"content-type": "application/json", "x-request-id": "6aacf17c-aa94-4935-975e-0596cd181bae", "content-length": "29", "date": "Wed, 30 Apr 2025 06:43:06 GMT"} })
102432025-04-30T06:43:07.749ZTRACtest_snapshot_basic (oximeter-agent): ignoring request to update existing metric producer, since the endpoint information is the same as the existing collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = 354a52d4-3078-4706-81ce-12c32f6bbeaa
102442025-04-30T06:43:07.749ZTRACtest_snapshot_basic (oximeter-agent): ignoring request to update existing metric producer, since the endpoint information is the same as the existing collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
102452025-04-30T06:43:07.749ZINFOtest_snapshot_basic (oximeter-agent): refreshed list of producers from Nexus collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 file = oximeter/collector/src/agent.rs:637 n_current_tasks = 2 n_pruned_tasks = 0
102462025-04-30T06:43:07.750ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "79835900-9ab6-4f9b-9d63-74c75f701902", "content-length": "170", "date": "Wed, 30 Apr 2025 06:43:06 GMT"} })
102472025-04-30T06:43:07.751ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/0/component/stage0/caboose?firmware_slot=0
102482025-04-30T06:43:07.754ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "57cff411-df54-4b7a-9ab6-21a8181833fe", "content-length": "179", "date": "Wed, 30 Apr 2025 06:43:06 GMT"} })
102492025-04-30T06:43:07.754ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/0/component/stage0/caboose?firmware_slot=1
102502025-04-30T06:43:07.759ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "75b51516-9cde-4132-b981-6ed8db01a3ef", "content-length": "179", "date": "Wed, 30 Apr 2025 06:43:06 GMT"} })
102512025-04-30T06:43:07.759ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/0/component/rot/cmpa
102522025-04-30T06:43:07.760ZDEBGtest_snapshot_basic: attempt to fetch latest inventory collection result = Ok("found")
102532025-04-30T06:43:07.761ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "5bb306e9-f8a0-4653-851d-694be7f90858", "content-length": "702", "date": "Wed, 30 Apr 2025 06:43:06 GMT"} })
102542025-04-30T06:43:07.761ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/0/component/rot/cfpa
102552025-04-30T06:43:07.763ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "f45313a6-12c9-43e2-8a1f-5c1581928348", "content-length": "727", "date": "Wed, 30 Apr 2025 06:43:06 GMT"} })
102562025-04-30T06:43:07.763ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/0/component/rot/cfpa
102572025-04-30T06:43:07.766ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "7a597358-12bf-4227-ba46-02f501ab22f0", "content-length": "729", "date": "Wed, 30 Apr 2025 06:43:06 GMT"} })
102582025-04-30T06:43:07.766ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/0/component/rot/cfpa
102592025-04-30T06:43:07.767ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "7f610b82-fcd2-4367-a9b1-7e59a14ee630", "content-length": "728", "date": "Wed, 30 Apr 2025 06:43:06 GMT"} })
102602025-04-30T06:43:07.767ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/1
102612025-04-30T06:43:07.770ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "b07fa716-91b8-47c5-959e-9fdf1722bf08", "content-length": "734", "date": "Wed, 30 Apr 2025 06:43:06 GMT"} })
102622025-04-30T06:43:07.771ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/1/component/sp/caboose?firmware_slot=0
102632025-04-30T06:43:07.789ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "2ef94987-cb1e-4412-b8ff-6d89b48cf6d7", "content-length": "111", "date": "Wed, 30 Apr 2025 06:43:06 GMT"} })
102642025-04-30T06:43:07.789ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/1/component/sp/caboose?firmware_slot=1
102652025-04-30T06:43:07.791ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "cf72a1b5-af6c-49b2-9825-637eeec230cf", "content-length": "111", "date": "Wed, 30 Apr 2025 06:43:07 GMT"} })
102662025-04-30T06:43:07.791ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/1/component/rot/caboose?firmware_slot=0
102672025-04-30T06:43:07.793ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "a7a8a13f-a5f6-40b9-8b6c-7a17d90ac53d", "content-length": "170", "date": "Wed, 30 Apr 2025 06:43:07 GMT"} })
102682025-04-30T06:43:07.793ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/1/component/rot/caboose?firmware_slot=1
102692025-04-30T06:43:07.795ZDEBGtest_snapshot_basic (collection-task): sent timer-based collection request to the collection task collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
102702025-04-30T06:43:07.795ZDEBGtest_snapshot_basic (collection-task): collecting from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
102712025-04-30T06:43:07.803ZTRACtest_snapshot_basic (collection-task): sent collection request to producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
102722025-04-30T06:43:07.804ZDEBGtest_snapshot_basic (collection-task): collected results from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 n_results = 1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
102732025-04-30T06:43:07.804ZTRACtest_snapshot_basic (collection-task): forwarded results to main collection loop collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
102742025-04-30T06:43:07.804ZTRACtest_snapshot_basic (collection-task): top of inner collection loop, waiting for next request collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
102752025-04-30T06:43:07.806ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "a1e3da56-ce6a-469f-973b-38d2e1d2a952", "content-length": "170", "date": "Wed, 30 Apr 2025 06:43:07 GMT"} })
102762025-04-30T06:43:07.806ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/1/component/stage0/caboose?firmware_slot=0
102772025-04-30T06:43:07.811ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
102782025-04-30T06:43:07.811ZTRACtest_snapshot_basic (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
102792025-04-30T06:43:07.811ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
102802025-04-30T06:43:07.811ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {} }
102812025-04-30T06:43:07.813ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
102822025-04-30T06:43:07.815ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
102832025-04-30T06:43:07.821ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
102842025-04-30T06:43:07.821ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
102852025-04-30T06:43:07.821ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {} }
102862025-04-30T06:43:07.822ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
102872025-04-30T06:43:07.825ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
102882025-04-30T06:43:07.825ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {} }
102892025-04-30T06:43:07.829ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
102902025-04-30T06:43:07.829ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "7544f9f4-8ec4-41a2-9425-c77621d0e863", "content-length": "179", "date": "Wed, 30 Apr 2025 06:43:07 GMT"} })
102912025-04-30T06:43:07.829ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/1/component/stage0/caboose?firmware_slot=1
102922025-04-30T06:43:07.833ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "6b3f2766-ef95-4ff3-9e8a-40f51f3ec199", "content-length": "179", "date": "Wed, 30 Apr 2025 06:43:07 GMT"} })
102932025-04-30T06:43:07.834ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/1/component/rot/cmpa
102942025-04-30T06:43:07.835ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "8156260b-03fe-45b3-877b-503a8d3a2433", "content-length": "702", "date": "Wed, 30 Apr 2025 06:43:07 GMT"} })
102952025-04-30T06:43:07.835ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/1/component/rot/cfpa
102962025-04-30T06:43:07.844ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "842e85e5-ef74-43a8-b2ec-0fe9c16f2184", "content-length": "727", "date": "Wed, 30 Apr 2025 06:43:07 GMT"} })
102972025-04-30T06:43:07.844ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/1/component/rot/cfpa
102982025-04-30T06:43:07.848ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "2f79b6b8-f27a-4e0a-b02d-450151e7a3a2", "content-length": "729", "date": "Wed, 30 Apr 2025 06:43:07 GMT"} })
102992025-04-30T06:43:07.850ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/1/component/rot/cfpa
103002025-04-30T06:43:07.852ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "b1e03f35-b264-44d7-8afb-e327ab87e62b", "content-length": "728", "date": "Wed, 30 Apr 2025 06:43:07 GMT"} })
103012025-04-30T06:43:07.852ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/0
103022025-04-30T06:43:07.855ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "d1b80805-a03e-46d6-abb0-654043a23ef0", "content-length": "723", "date": "Wed, 30 Apr 2025 06:43:07 GMT"} })
103032025-04-30T06:43:07.855ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/0/component/sp/caboose?firmware_slot=0
103042025-04-30T06:43:07.859ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "d907b8e9-4db6-4037-afbc-541311dca34f", "content-length": "109", "date": "Wed, 30 Apr 2025 06:43:07 GMT"} })
103052025-04-30T06:43:07.859ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/0/component/sp/caboose?firmware_slot=1
103062025-04-30T06:43:07.867ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "8f531c76-0570-4b41-8b98-872a8f311eb7", "content-length": "109", "date": "Wed, 30 Apr 2025 06:43:07 GMT"} })
103072025-04-30T06:43:07.867ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/0/component/rot/caboose?firmware_slot=0
103082025-04-30T06:43:07.870ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "8920d778-e4f3-42bd-bdf8-951f1e94778e", "content-length": "169", "date": "Wed, 30 Apr 2025 06:43:07 GMT"} })
103092025-04-30T06:43:07.870ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/0/component/rot/caboose?firmware_slot=1
103102025-04-30T06:43:07.892ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "22647adc-65dd-445e-8a59-19a65a13d641", "content-length": "169", "date": "Wed, 30 Apr 2025 06:43:07 GMT"} })
103112025-04-30T06:43:07.893ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/0/component/stage0/caboose?firmware_slot=0
103122025-04-30T06:43:07.893ZDEBGtest_snapshot_basic: attempt to fetch latest inventory collection result = Ok("found")
103132025-04-30T06:43:07.895ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "763bf552-4cf8-42d6-b61f-03762eb6d6b7", "content-length": "178", "date": "Wed, 30 Apr 2025 06:43:07 GMT"} })
103142025-04-30T06:43:07.895ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/0/component/stage0/caboose?firmware_slot=1
103152025-04-30T06:43:07.903ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "c2565283-098e-422f-ac12-9f423c754a35", "content-length": "178", "date": "Wed, 30 Apr 2025 06:43:07 GMT"} })
103162025-04-30T06:43:07.903ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/0/component/rot/cmpa
103172025-04-30T06:43:07.906ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "0c25f3ba-ba59-4841-8ee1-3500f0247aa8", "content-length": "702", "date": "Wed, 30 Apr 2025 06:43:07 GMT"} })
103182025-04-30T06:43:07.906ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/0/component/rot/cfpa
103192025-04-30T06:43:07.908ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "8615a97f-10f7-4945-8b63-e94f30b2cbf2", "content-length": "727", "date": "Wed, 30 Apr 2025 06:43:07 GMT"} })
103202025-04-30T06:43:07.909ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/0/component/rot/cfpa
103212025-04-30T06:43:07.912ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "84bd7ffb-e8af-45d9-9b83-fff6cdf2968c", "content-length": "729", "date": "Wed, 30 Apr 2025 06:43:07 GMT"} })
103222025-04-30T06:43:07.912ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/0/component/rot/cfpa
103232025-04-30T06:43:07.913ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "ebea6415-a2ca-4fc6-8aa3-d4566b72a51a", "content-length": "728", "date": "Wed, 30 Apr 2025 06:43:07 GMT"} })
103242025-04-30T06:43:07.913ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/1
103252025-04-30T06:43:07.915ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "754e83d7-b4f1-460a-90d3-b7c612f41bdd", "content-length": "723", "date": "Wed, 30 Apr 2025 06:43:07 GMT"} })
103262025-04-30T06:43:07.915ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/1/component/sp/caboose?firmware_slot=0
103272025-04-30T06:43:07.917ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "e110925c-a4f6-43dc-a777-8fb299c546d5", "content-length": "109", "date": "Wed, 30 Apr 2025 06:43:07 GMT"} })
103282025-04-30T06:43:07.917ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/1/component/sp/caboose?firmware_slot=1
103292025-04-30T06:43:07.919ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "08fcbe32-69ff-45f0-8193-29bc0c93b6eb", "content-length": "109", "date": "Wed, 30 Apr 2025 06:43:07 GMT"} })
103302025-04-30T06:43:07.919ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/1/component/rot/caboose?firmware_slot=0
103312025-04-30T06:43:07.922ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "c4c96c9a-309e-4b01-be10-fd4535a4f736", "content-length": "169", "date": "Wed, 30 Apr 2025 06:43:07 GMT"} })
103322025-04-30T06:43:07.922ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/1/component/rot/caboose?firmware_slot=1
103332025-04-30T06:43:07.927ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "381913d8-ff16-4bca-adec-f009f9d53bb4", "content-length": "169", "date": "Wed, 30 Apr 2025 06:43:07 GMT"} })
103342025-04-30T06:43:07.930ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/1/component/stage0/caboose?firmware_slot=0
103352025-04-30T06:43:07.934ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "4286d06a-1d68-4a5d-85f3-659599aaa1ae", "content-length": "178", "date": "Wed, 30 Apr 2025 06:43:07 GMT"} })
103362025-04-30T06:43:07.934ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/1/component/stage0/caboose?firmware_slot=1
103372025-04-30T06:43:07.938ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "fa46341d-ebc2-4dc2-8354-f040302ee760", "content-length": "178", "date": "Wed, 30 Apr 2025 06:43:07 GMT"} })
103382025-04-30T06:43:07.938ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/1/component/rot/cmpa
103392025-04-30T06:43:07.939ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "a27e18cc-daf6-43d8-b607-39a3f57ff2b1", "content-length": "702", "date": "Wed, 30 Apr 2025 06:43:07 GMT"} })
103402025-04-30T06:43:07.939ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/1/component/rot/cfpa
103412025-04-30T06:43:07.940ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "8bec445e-6808-49a8-ae9d-62baa17ac1fa", "content-length": "727", "date": "Wed, 30 Apr 2025 06:43:07 GMT"} })
103422025-04-30T06:43:07.940ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/1/component/rot/cfpa
103432025-04-30T06:43:07.941ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "ffcd9335-41e3-40f8-be78-664ff2b401f5", "content-length": "729", "date": "Wed, 30 Apr 2025 06:43:07 GMT"} })
103442025-04-30T06:43:07.942ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/1/component/rot/cfpa
103452025-04-30T06:43:07.943ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "b358a8db-fa0b-4831-9c62-796489703a40", "content-length": "728", "date": "Wed, 30 Apr 2025 06:43:07 GMT"} })
103462025-04-30T06:43:07.943ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
103472025-04-30T06:43:07.943ZTRACtest_snapshot_basic (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
103482025-04-30T06:43:07.943ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
103492025-04-30T06:43:07.943ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
103502025-04-30T06:43:07.944ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
103512025-04-30T06:43:07.944ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
103522025-04-30T06:43:07.944ZTRACtest_snapshot_basic (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
103532025-04-30T06:43:07.944ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
103542025-04-30T06:43:07.944ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {} }
103552025-04-30T06:43:07.944ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
103562025-04-30T06:43:07.946ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
103572025-04-30T06:43:07.950ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
103582025-04-30T06:43:07.950ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
103592025-04-30T06:43:07.950ZTRACtest_snapshot_basic (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
103602025-04-30T06:43:07.950ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
103612025-04-30T06:43:07.950ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
103622025-04-30T06:43:07.951ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
103632025-04-30T06:43:07.951ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
103642025-04-30T06:43:07.958ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
103652025-04-30T06:43:07.959ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
103662025-04-30T06:43:07.959ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {} }
103672025-04-30T06:43:07.960ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
103682025-04-30T06:43:07.962ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
103692025-04-30T06:43:07.969ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
103702025-04-30T06:43:07.969ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
103712025-04-30T06:43:07.969ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
103722025-04-30T06:43:07.969ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
103732025-04-30T06:43:07.972ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
103742025-04-30T06:43:07.972ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {} }
103752025-04-30T06:43:07.973ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
103762025-04-30T06:43:08.045ZDEBGtest_snapshot_basic (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:41923
103772025-04-30T06:43:08.045ZDEBGtest_snapshot_basic (BackgroundTasks): client request SledAgent = http://[::1]:41923 background_task = inventory_collection body = None method = GET uri = http://[::1]:41923/inventory
103782025-04-30T06:43:08.046ZINFOtest_snapshot_basic (dropshot): accepted connection local_addr = [::1]:41923 remote_addr = [::1]:35462 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
103792025-04-30T06:43:08.046ZDEBGtest_snapshot_basic (results-sink): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
103802025-04-30T06:43:08.046ZDEBGtest_snapshot_basic (results-sink-cluster): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
103812025-04-30T06:43:08.047ZDEBGtest_snapshot_basic (clickhouse-client): unrolling 1 total samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 6ad06c6b-9a6e-4ed1-b3f4-5d2e9da3f8a7
103822025-04-30T06:43:08.048ZTRACtest_snapshot_basic (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 6ad06c6b-9a6e-4ed1-b3f4-5d2e9da3f8a7 n_columns = 4 n_rows = 2 sql = INSERT INTO oximeter.fields_string FORMAT Native
103832025-04-30T06:43:08.052ZTRACtest_snapshot_basic (dropshot): incoming request local_addr = [::1]:41923 method = GET remote_addr = [::1]:35462 req_id = a90c958c-5d0f-4b90-8b60-0dbb195c1ddf sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
103842025-04-30T06:43:08.053ZINFOtest_snapshot_basic (dropshot): request completed latency_us = 558 local_addr = [::1]:41923 method = GET remote_addr = [::1]:35462 req_id = a90c958c-5d0f-4b90-8b60-0dbb195c1ddf response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
103852025-04-30T06:43:08.054ZDEBGtest_snapshot_basic (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 6ad06c6b-9a6e-4ed1-b3f4-5d2e9da3f8a7 n_rows = 2 table_name = fields_string
103862025-04-30T06:43:08.054ZTRACtest_snapshot_basic (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 6ad06c6b-9a6e-4ed1-b3f4-5d2e9da3f8a7 n_columns = 4 n_rows = 1 sql = INSERT INTO oximeter.measurements_i64 FORMAT Native
103872025-04-30T06:43:08.055ZDEBGtest_snapshot_basic (BackgroundTasks): client response SledAgent = http://[::1]:41923 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:41923/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "a90c958c-5d0f-4b90-8b60-0dbb195c1ddf", "content-length": "3141", "date": "Wed, 30 Apr 2025 06:43:07 GMT"} })
103882025-04-30T06:43:08.059ZDEBGtest_snapshot_basic (BackgroundTasks): begin collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
103892025-04-30T06:43:08.059ZDEBGtest_snapshot_basic (BackgroundTasks): end collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
103902025-04-30T06:43:08.059ZDEBGtest_snapshot_basic (BackgroundTasks): finished collection background_task = inventory_collection
103912025-04-30T06:43:08.063ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
103922025-04-30T06:43:08.063ZTRACtest_snapshot_basic (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
103932025-04-30T06:43:08.063ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
103942025-04-30T06:43:08.063ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
103952025-04-30T06:43:08.064ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
103962025-04-30T06:43:08.074ZDEBGtest_snapshot_basic (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 6ad06c6b-9a6e-4ed1-b3f4-5d2e9da3f8a7 n_rows = 1 table_name = measurements_i64
103972025-04-30T06:43:08.075ZTRACtest_snapshot_basic (results-sink): successfully inserted samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
103982025-04-30T06:43:08.075ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
103992025-04-30T06:43:08.084ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
104002025-04-30T06:43:08.085ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
104012025-04-30T06:43:08.086ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
104022025-04-30T06:43:08.087ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
104032025-04-30T06:43:08.121ZDEBGtest_snapshot_basic: attempt to fetch latest inventory collection result = Ok("found")
104042025-04-30T06:43:08.131ZWARNtest_snapshot_basic (datastore): Non-retryable transaction failure err = DatabaseError(UniqueViolation, "duplicate key value violates unique constraint \\"inv_caboose_pkey\\"") transaction = inventory_insert_collection
104052025-04-30T06:43:08.132ZWARNtest_snapshot_basic (BackgroundTasks): inventory collection failed background_task = inventory_collection error = failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint "inv_caboose_pkey"
104062025-04-30T06:43:08.132ZDEBGtest_snapshot_basic (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 638.788655ms iteration = 36 status = {"error":"failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint \\"inv_caboose_pkey\\""}
104072025-04-30T06:43:08.132ZDEBGtest_snapshot_basic (BackgroundTasks): activating background_task = inventory_collection iteration = 37 reason = Signaled
104082025-04-30T06:43:08.132ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
104092025-04-30T06:43:08.132ZTRACtest_snapshot_basic (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
104102025-04-30T06:43:08.132ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
104112025-04-30T06:43:08.132ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
104122025-04-30T06:43:08.133ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
104132025-04-30T06:43:08.136ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
104142025-04-30T06:43:08.140ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
104152025-04-30T06:43:08.140ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
104162025-04-30T06:43:08.140ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
104172025-04-30T06:43:08.141ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
104182025-04-30T06:43:08.144ZDEBGtest_snapshot_basic (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = [(d1ad44e5-18dd-4193-a70e-bf9042f0e0e1, 0)]
104192025-04-30T06:43:08.144ZTRACtest_snapshot_basic (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
104202025-04-30T06:43:08.145ZTRACtest_snapshot_basic (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: 5e 88 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 ^............_mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:50597 req_id = f477e50e-619c-4e33-be40-588bed736041
104212025-04-30T06:43:08.145ZDEBGtest_snapshot_basic (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 24200,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:50597 req_id = f477e50e-619c-4e33-be40-588bed736041
104222025-04-30T06:43:08.145ZDEBGtest_snapshot_basic (store): query key key = _mgs._tcp
104232025-04-30T06:43:08.146ZDEBGtest_snapshot_basic (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
104242025-04-30T06:43:08.146ZDEBGtest_snapshot_basic (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:50597 query = LowerQuery { name: LowerName(Name("_mgs._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 41502, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = f477e50e-619c-4e33-be40-588bed736041
104252025-04-30T06:43:08.146ZDEBGtest_snapshot_basic (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 41502, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 4252, tv_nsec: 945413916 } })
104262025-04-30T06:43:08.146ZTRACtest_snapshot_basic (DnsResolver): lookup_service_targets: looking up SRV target
104272025-04-30T06:43:08.147ZTRACtest_snapshot_basic (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: 06 1f 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e ............-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:53645 req_id = 37fab316-2ebe-46cf-b8ca-345a6e3b9926
104282025-04-30T06:43:08.147ZDEBGtest_snapshot_basic (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 1567,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:53645 req_id = 37fab316-2ebe-46cf-b8ca-345a6e3b9926
104292025-04-30T06:43:08.147ZDEBGtest_snapshot_basic (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
104302025-04-30T06:43:08.147ZDEBGtest_snapshot_basic (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:53645 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = 37fab316-2ebe-46cf-b8ca-345a6e3b9926
104312025-04-30T06:43:08.210ZTRACtest_snapshot_basic (DnsResolver): lookup_all_socket_v6 srv dns_name = _clickhouse-admin-keeper._tcp.control-plane.oxide.internal
104322025-04-30T06:43:08.210ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
104332025-04-30T06:43:08.210ZTRACtest_snapshot_basic (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
104342025-04-30T06:43:08.210ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
104352025-04-30T06:43:08.210ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {} }
104362025-04-30T06:43:08.211ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
104372025-04-30T06:43:08.211ZWARNtest_snapshot_basic (results-sink-cluster): failed to insert some results into metric DB collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 error = Failed to check out connection to database: No backends found for this service file = oximeter/collector/src/results_sink.rs:92
104382025-04-30T06:43:08.211ZTRACtest_snapshot_basic (dns): buffer buffer = Length: 87 (0x57) bytes\n0000: db 77 01 00 00 01 00 00 00 00 00 01 18 5f 63 6c .w..........._cl\n0010: 69 63 6b 68 6f 75 73 65 2d 61 64 6d 69 6e 2d 6b ickhouse-admin-k\n0020: 65 65 70 65 72 04 5f 74 63 70 0d 63 6f 6e 74 72 eeper._tcp.contr\n0030: 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 69 ol-plane.oxide.i\n0040: 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 04 nternal..!....).\n0050: d0 00 00 00 00 00 00 ....... kind = dns peer_addr = [::1]:62401 req_id = 10affdb6-4782-4696-8337-801d41eecff2
104392025-04-30T06:43:08.211ZDEBGtest_snapshot_basic (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 56183,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 24,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 97,\n 100,\n 109,\n 105,\n 110,\n 45,\n 107,\n 101,\n 101,\n 112,\n 101,\n 114,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:62401 req_id = 10affdb6-4782-4696-8337-801d41eecff2
104402025-04-30T06:43:08.211ZDEBGtest_snapshot_basic (store): query key key = _clickhouse-admin-keeper._tcp
104412025-04-30T06:43:08.212ZERROtest_snapshot_basic (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 56183, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } NXDOMAIN: no records found for name: "_clickhouse-admin-keeper._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:62401 req_id = 10affdb6-4782-4696-8337-801d41eecff2
104422025-04-30T06:43:08.213ZDEBGtest_snapshot_basic (BackgroundTasks): begin collection background_task = inventory_collection
104432025-04-30T06:43:08.213ZDEBGtest_snapshot_basic (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:41502
104442025-04-30T06:43:08.213ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/ignition
104452025-04-30T06:43:08.216ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
104462025-04-30T06:43:08.221ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
104472025-04-30T06:43:08.221ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
104482025-04-30T06:43:08.221ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {} }
104492025-04-30T06:43:08.222ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
104502025-04-30T06:43:08.222ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "97a1f2b6-7bc7-4cf4-8fbc-1e330b26a323", "content-length": "835", "date": "Wed, 30 Apr 2025 06:43:07 GMT"} })
104512025-04-30T06:43:08.222ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/0
104522025-04-30T06:43:08.225ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
104532025-04-30T06:43:08.225ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {} }
104542025-04-30T06:43:08.225ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
104552025-04-30T06:43:08.226ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "460e2192-c445-4f60-ab2d-d3f0ca4160c9", "content-length": "734", "date": "Wed, 30 Apr 2025 06:43:07 GMT"} })
104562025-04-30T06:43:08.226ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/0/component/sp/caboose?firmware_slot=0
104572025-04-30T06:43:08.236ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "632a5b9e-97d1-4579-bd83-7c80785ba704", "content-length": "111", "date": "Wed, 30 Apr 2025 06:43:07 GMT"} })
104582025-04-30T06:43:08.236ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/0/component/sp/caboose?firmware_slot=1
104592025-04-30T06:43:08.239ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "9bada0d8-ff69-4772-ac84-c67e95aedec3", "content-length": "111", "date": "Wed, 30 Apr 2025 06:43:07 GMT"} })
104602025-04-30T06:43:08.239ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/0/component/rot/caboose?firmware_slot=0
104612025-04-30T06:43:08.241ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "01cc9ac6-47ae-4e5b-86d3-f38e8502c250", "content-length": "170", "date": "Wed, 30 Apr 2025 06:43:07 GMT"} })
104622025-04-30T06:43:08.242ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/0/component/rot/caboose?firmware_slot=1
104632025-04-30T06:43:08.244ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "08f690da-60e2-4653-9cf1-6373750780ad", "content-length": "170", "date": "Wed, 30 Apr 2025 06:43:07 GMT"} })
104642025-04-30T06:43:08.244ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/0/component/stage0/caboose?firmware_slot=0
104652025-04-30T06:43:08.246ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "fae9566b-2962-42b1-b6e5-8e74e338070f", "content-length": "179", "date": "Wed, 30 Apr 2025 06:43:07 GMT"} })
104662025-04-30T06:43:08.246ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/0/component/stage0/caboose?firmware_slot=1
104672025-04-30T06:43:08.248ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "460ea8a6-c7f9-4d86-ba92-ae714e8392e8", "content-length": "179", "date": "Wed, 30 Apr 2025 06:43:07 GMT"} })
104682025-04-30T06:43:08.248ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/0/component/rot/cmpa
104692025-04-30T06:43:08.249ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "f361f5f6-772b-40ed-9360-fb0ddd83127b", "content-length": "702", "date": "Wed, 30 Apr 2025 06:43:07 GMT"} })
104702025-04-30T06:43:08.249ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/0/component/rot/cfpa
104712025-04-30T06:43:08.250ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "b69d7f41-7eca-4078-9182-bd2e2d517331", "content-length": "727", "date": "Wed, 30 Apr 2025 06:43:07 GMT"} })
104722025-04-30T06:43:08.251ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/0/component/rot/cfpa
104732025-04-30T06:43:08.252ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "abfab602-58a9-4f38-a7ff-3234e86abd92", "content-length": "729", "date": "Wed, 30 Apr 2025 06:43:07 GMT"} })
104742025-04-30T06:43:08.252ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/0/component/rot/cfpa
104752025-04-30T06:43:08.253ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "37a56b01-f9d6-4992-ad6e-bde38d0ae713", "content-length": "728", "date": "Wed, 30 Apr 2025 06:43:07 GMT"} })
104762025-04-30T06:43:08.254ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/1
104772025-04-30T06:43:08.255ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "d5f28fa8-0d23-4a63-b72a-623a8ca5c950", "content-length": "734", "date": "Wed, 30 Apr 2025 06:43:07 GMT"} })
104782025-04-30T06:43:08.255ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/1/component/sp/caboose?firmware_slot=0
104792025-04-30T06:43:08.259ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "5757cb38-6bed-4f81-80f2-c38481be2c7b", "content-length": "111", "date": "Wed, 30 Apr 2025 06:43:07 GMT"} })
104802025-04-30T06:43:08.259ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/1/component/sp/caboose?firmware_slot=1
104812025-04-30T06:43:08.264ZDEBGtest_snapshot_basic: attempt to fetch latest inventory collection result = Ok("found")
104822025-04-30T06:43:08.265ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "156f4fa3-ed8d-4d01-9082-f0eefc442b9f", "content-length": "111", "date": "Wed, 30 Apr 2025 06:43:07 GMT"} })
104832025-04-30T06:43:08.266ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/1/component/rot/caboose?firmware_slot=0
104842025-04-30T06:43:08.268ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "e1df2b45-c9a1-4a3c-9380-1dad3241d085", "content-length": "170", "date": "Wed, 30 Apr 2025 06:43:07 GMT"} })
104852025-04-30T06:43:08.268ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/1/component/rot/caboose?firmware_slot=1
104862025-04-30T06:43:08.270ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "4f95aff5-a161-48b5-b37c-71c1088856f3", "content-length": "170", "date": "Wed, 30 Apr 2025 06:43:07 GMT"} })
104872025-04-30T06:43:08.270ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/1/component/stage0/caboose?firmware_slot=0
104882025-04-30T06:43:08.273ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "9be2d04a-febc-48b2-9357-3d5470ea58e8", "content-length": "179", "date": "Wed, 30 Apr 2025 06:43:07 GMT"} })
104892025-04-30T06:43:08.273ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/1/component/stage0/caboose?firmware_slot=1
104902025-04-30T06:43:08.276ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "d24a3aa4-0f7c-4573-bcdf-bf258811cbb7", "content-length": "179", "date": "Wed, 30 Apr 2025 06:43:07 GMT"} })
104912025-04-30T06:43:08.276ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/1/component/rot/cmpa
104922025-04-30T06:43:08.278ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "271ed4b5-2f83-40cf-9731-907d30bcacac", "content-length": "702", "date": "Wed, 30 Apr 2025 06:43:07 GMT"} })
104932025-04-30T06:43:08.278ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/1/component/rot/cfpa
104942025-04-30T06:43:08.280ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "f2bd63a2-6d09-45eb-bc0d-1fb0dc7a46af", "content-length": "727", "date": "Wed, 30 Apr 2025 06:43:07 GMT"} })
104952025-04-30T06:43:08.280ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/1/component/rot/cfpa
104962025-04-30T06:43:08.281ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "7a039557-7bac-4d4c-ac27-b2e53e7b0174", "content-length": "729", "date": "Wed, 30 Apr 2025 06:43:07 GMT"} })
104972025-04-30T06:43:08.281ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/1/component/rot/cfpa
104982025-04-30T06:43:08.282ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "d2faa38a-0b55-45b0-8678-3b19a93b5fc3", "content-length": "728", "date": "Wed, 30 Apr 2025 06:43:07 GMT"} })
104992025-04-30T06:43:08.282ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/0
105002025-04-30T06:43:08.285ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "f61bebae-cdf2-4903-97f5-4e26af8d254a", "content-length": "723", "date": "Wed, 30 Apr 2025 06:43:07 GMT"} })
105012025-04-30T06:43:08.285ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/0/component/sp/caboose?firmware_slot=0
105022025-04-30T06:43:08.287ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "34949c7f-b720-4a76-9995-ed3a0f4daed0", "content-length": "109", "date": "Wed, 30 Apr 2025 06:43:07 GMT"} })
105032025-04-30T06:43:08.287ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/0/component/sp/caboose?firmware_slot=1
105042025-04-30T06:43:08.290ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "37bbd484-205c-4175-9ae3-69dbe2ec659c", "content-length": "109", "date": "Wed, 30 Apr 2025 06:43:07 GMT"} })
105052025-04-30T06:43:08.290ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/0/component/rot/caboose?firmware_slot=0
105062025-04-30T06:43:08.292ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "6564cef9-0474-4a4d-b3f1-49df14794e39", "content-length": "169", "date": "Wed, 30 Apr 2025 06:43:07 GMT"} })
105072025-04-30T06:43:08.292ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/0/component/rot/caboose?firmware_slot=1
105082025-04-30T06:43:08.296ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "6c293e88-b83b-4af9-8b38-c53f28e2e2c5", "content-length": "169", "date": "Wed, 30 Apr 2025 06:43:07 GMT"} })
105092025-04-30T06:43:08.296ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/0/component/stage0/caboose?firmware_slot=0
105102025-04-30T06:43:08.298ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "b5674dc0-7fda-42aa-960a-44b060da9968", "content-length": "178", "date": "Wed, 30 Apr 2025 06:43:07 GMT"} })
105112025-04-30T06:43:08.298ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/0/component/stage0/caboose?firmware_slot=1
105122025-04-30T06:43:08.300ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "5fa42edf-d8ff-4937-bc0f-de6f7f9829e9", "content-length": "178", "date": "Wed, 30 Apr 2025 06:43:07 GMT"} })
105132025-04-30T06:43:08.301ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/0/component/rot/cmpa
105142025-04-30T06:43:08.302ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "9fc5c588-a807-468d-af71-d6d4ee0ab951", "content-length": "702", "date": "Wed, 30 Apr 2025 06:43:07 GMT"} })
105152025-04-30T06:43:08.302ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/0/component/rot/cfpa
105162025-04-30T06:43:08.303ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "da49a886-0937-41ed-adee-ff8be120a23e", "content-length": "727", "date": "Wed, 30 Apr 2025 06:43:07 GMT"} })
105172025-04-30T06:43:08.303ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/0/component/rot/cfpa
105182025-04-30T06:43:08.304ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "a6eeddf8-c371-408c-b225-02c49bb19e28", "content-length": "729", "date": "Wed, 30 Apr 2025 06:43:07 GMT"} })
105192025-04-30T06:43:08.304ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/0/component/rot/cfpa
105202025-04-30T06:43:08.305ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "3c48b3f7-b127-4887-89e3-49b4efc8e7b0", "content-length": "728", "date": "Wed, 30 Apr 2025 06:43:07 GMT"} })
105212025-04-30T06:43:08.306ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/1
105222025-04-30T06:43:08.308ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "e51dc69f-9628-4e74-af36-90e82917e11c", "content-length": "723", "date": "Wed, 30 Apr 2025 06:43:07 GMT"} })
105232025-04-30T06:43:08.308ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/1/component/sp/caboose?firmware_slot=0
105242025-04-30T06:43:08.310ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "b50f2acf-a85d-49d5-9528-488162afbd0b", "content-length": "109", "date": "Wed, 30 Apr 2025 06:43:07 GMT"} })
105252025-04-30T06:43:08.310ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/1/component/sp/caboose?firmware_slot=1
105262025-04-30T06:43:08.311ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "5e33fc08-f818-4756-a375-3353a9521ebe", "content-length": "109", "date": "Wed, 30 Apr 2025 06:43:07 GMT"} })
105272025-04-30T06:43:08.312ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/1/component/rot/caboose?firmware_slot=0
105282025-04-30T06:43:08.313ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "433a734e-ddca-423d-9856-6416b56fc0d8", "content-length": "169", "date": "Wed, 30 Apr 2025 06:43:07 GMT"} })
105292025-04-30T06:43:08.314ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/1/component/rot/caboose?firmware_slot=1
105302025-04-30T06:43:08.315ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
105312025-04-30T06:43:08.315ZTRACtest_snapshot_basic (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
105322025-04-30T06:43:08.315ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
105332025-04-30T06:43:08.315ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {} }
105342025-04-30T06:43:08.317ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
105352025-04-30T06:43:08.320ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
105362025-04-30T06:43:08.328ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
105372025-04-30T06:43:08.328ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
105382025-04-30T06:43:08.328ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {} }
105392025-04-30T06:43:08.328ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
105402025-04-30T06:43:08.329ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "68887a56-e5b7-48f1-ae71-7a2e9cd4a377", "content-length": "169", "date": "Wed, 30 Apr 2025 06:43:07 GMT"} })
105412025-04-30T06:43:08.329ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/1/component/stage0/caboose?firmware_slot=0
105422025-04-30T06:43:08.332ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
105432025-04-30T06:43:08.332ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {} }
105442025-04-30T06:43:08.332ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
105452025-04-30T06:43:08.337ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "a278b140-085f-4a76-abe6-189356c8e5db", "content-length": "178", "date": "Wed, 30 Apr 2025 06:43:07 GMT"} })
105462025-04-30T06:43:08.337ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/1/component/stage0/caboose?firmware_slot=1
105472025-04-30T06:43:08.340ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "ff25b0d2-3729-4598-a221-7b82253efe82", "content-length": "178", "date": "Wed, 30 Apr 2025 06:43:07 GMT"} })
105482025-04-30T06:43:08.340ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/1/component/rot/cmpa
105492025-04-30T06:43:08.342ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "42075934-5958-47a6-897e-e099fd6f9d53", "content-length": "702", "date": "Wed, 30 Apr 2025 06:43:07 GMT"} })
105502025-04-30T06:43:08.342ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/1/component/rot/cfpa
105512025-04-30T06:43:08.343ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "d0829518-95ee-4fe5-ad1a-aeeccf109459", "content-length": "727", "date": "Wed, 30 Apr 2025 06:43:07 GMT"} })
105522025-04-30T06:43:08.344ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/1/component/rot/cfpa
105532025-04-30T06:43:08.345ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "8a4ee4de-4d49-4726-8acf-8f05bb0a5035", "content-length": "729", "date": "Wed, 30 Apr 2025 06:43:07 GMT"} })
105542025-04-30T06:43:08.345ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/1/component/rot/cfpa
105552025-04-30T06:43:08.346ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "4131b00f-ccb0-421e-be13-1c89ab7195e3", "content-length": "728", "date": "Wed, 30 Apr 2025 06:43:07 GMT"} })
105562025-04-30T06:43:08.346ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
105572025-04-30T06:43:08.346ZTRACtest_snapshot_basic (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
105582025-04-30T06:43:08.346ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
105592025-04-30T06:43:08.346ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
105602025-04-30T06:43:08.347ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
105612025-04-30T06:43:08.350ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
105622025-04-30T06:43:08.354ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
105632025-04-30T06:43:08.354ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
105642025-04-30T06:43:08.354ZTRACtest_snapshot_basic (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
105652025-04-30T06:43:08.354ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
105662025-04-30T06:43:08.354ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
105672025-04-30T06:43:08.355ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
105682025-04-30T06:43:08.363ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
105692025-04-30T06:43:08.366ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
105702025-04-30T06:43:08.367ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
105712025-04-30T06:43:08.367ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
105722025-04-30T06:43:08.369ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
105732025-04-30T06:43:08.454ZDEBGtest_snapshot_basic (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:41923
105742025-04-30T06:43:08.454ZDEBGtest_snapshot_basic (BackgroundTasks): client request SledAgent = http://[::1]:41923 background_task = inventory_collection body = None method = GET uri = http://[::1]:41923/inventory
105752025-04-30T06:43:08.456ZINFOtest_snapshot_basic (dropshot): accepted connection local_addr = [::1]:41923 remote_addr = [::1]:54108 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
105762025-04-30T06:43:08.456ZDEBGtest_snapshot_basic (BackgroundTasks): activating background_task = nat_v4_garbage_collector iteration = 2 reason = Timeout
105772025-04-30T06:43:08.457ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database
105782025-04-30T06:43:08.457ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
105792025-04-30T06:43:08.467ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
105802025-04-30T06:43:08.467ZDEBGtest_snapshot_basic (BackgroundTasks): activating background_task = bfd_manager iteration = 2 reason = Timeout
105812025-04-30T06:43:08.467ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = bfd_manager resource = Database
105822025-04-30T06:43:08.467ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = bfd_manager roles = RoleSet { roles: {} }
105832025-04-30T06:43:08.467ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = bfd_manager resource = Database result = Ok(())
105842025-04-30T06:43:08.468ZDEBGtest_snapshot_basic (BackgroundTasks): activating background_task = phantom_disks iteration = 2 reason = Timeout
105852025-04-30T06:43:08.468ZDEBGtest_snapshot_basic (BackgroundTasks): activating background_task = physical_disk_adoption iteration = 3 reason = Timeout
105862025-04-30T06:43:08.468ZDEBGtest_snapshot_basic (BackgroundTasks): activation complete background_task = physical_disk_adoption elapsed = 624.635\u{b5}s iteration = 3 status = {"error":"task disabled"}
105872025-04-30T06:43:08.469ZDEBGtest_snapshot_basic (BackgroundTasks): activating background_task = service_zone_nat_tracker iteration = 2 reason = Timeout
105882025-04-30T06:43:08.469ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Inventory
105892025-04-30T06:43:08.469ZTRACtest_snapshot_basic (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = service_zone_nat_tracker resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
105902025-04-30T06:43:08.469ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database
105912025-04-30T06:43:08.469ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
105922025-04-30T06:43:08.469ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
105932025-04-30T06:43:08.470ZDEBGtest_snapshot_basic (BackgroundTasks): activating background_task = switch_port_config_manager iteration = 2 reason = Timeout
105942025-04-30T06:43:08.470ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Fleet
105952025-04-30T06:43:08.470ZTRACtest_snapshot_basic (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = switch_port_config_manager resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
105962025-04-30T06:43:08.470ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database
105972025-04-30T06:43:08.470ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
105982025-04-30T06:43:08.471ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
105992025-04-30T06:43:08.474ZTRACtest_snapshot_basic (dropshot): incoming request local_addr = [::1]:41923 method = GET remote_addr = [::1]:54108 req_id = 70d44ba0-446d-4f75-b592-17e061835768 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
106002025-04-30T06:43:08.475ZINFOtest_snapshot_basic (dropshot): request completed latency_us = 1077 local_addr = [::1]:41923 method = GET remote_addr = [::1]:54108 req_id = 70d44ba0-446d-4f75-b592-17e061835768 response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
106012025-04-30T06:43:08.477ZDEBGtest_snapshot_basic: attempt to fetch latest inventory collection result = Ok("found")
106022025-04-30T06:43:08.478ZDEBGtest_snapshot_basic (BackgroundTasks): client response SledAgent = http://[::1]:41923 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:41923/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "70d44ba0-446d-4f75-b592-17e061835768", "content-length": "3141", "date": "Wed, 30 Apr 2025 06:43:07 GMT"} })
106032025-04-30T06:43:08.486ZDEBGtest_snapshot_basic (BackgroundTasks): begin collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
106042025-04-30T06:43:08.487ZDEBGtest_snapshot_basic (BackgroundTasks): end collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
106052025-04-30T06:43:08.487ZDEBGtest_snapshot_basic (BackgroundTasks): finished collection background_task = inventory_collection
106062025-04-30T06:43:08.507ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
106072025-04-30T06:43:08.508ZTRACtest_snapshot_basic (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
106082025-04-30T06:43:08.508ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
106092025-04-30T06:43:08.510ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
106102025-04-30T06:43:08.513ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
106112025-04-30T06:43:08.515ZTRACtest_snapshot_basic (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal
106122025-04-30T06:43:08.519ZTRACtest_snapshot_basic (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal
106132025-04-30T06:43:08.522ZDEBGtest_snapshot_basic (BackgroundTasks): activation complete background_task = phantom_disks elapsed = 54.362995ms iteration = 2 status = {"phantom_disk_deleted_err":0,"phantom_disk_deleted_ok":0}
106142025-04-30T06:43:08.523ZDEBGtest_snapshot_basic (BackgroundTasks): activating background_task = v2p_manager iteration = 2 reason = Timeout
106152025-04-30T06:43:08.523ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database
106162025-04-30T06:43:08.523ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
106172025-04-30T06:43:08.524ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
106182025-04-30T06:43:08.524ZDEBGtest_snapshot_basic (BackgroundTasks): activating background_task = instance_watcher iteration = 2 reason = Timeout
106192025-04-30T06:43:08.524ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Fleet
106202025-04-30T06:43:08.524ZTRACtest_snapshot_basic (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = instance_watcher resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
106212025-04-30T06:43:08.524ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database
106222025-04-30T06:43:08.524ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
106232025-04-30T06:43:08.525ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
106242025-04-30T06:43:08.525ZDEBGtest_snapshot_basic (BackgroundTasks): activating background_task = instance_updater iteration = 2 reason = Timeout
106252025-04-30T06:43:08.525ZINFOtest_snapshot_basic (BackgroundTasks): background instance updater explicitly disabled background_task = instance_updater
106262025-04-30T06:43:08.525ZDEBGtest_snapshot_basic (BackgroundTasks): activation complete background_task = instance_updater elapsed = 104.388\u{b5}s iteration = 2 status = {"destroyed_active_vmms":0,"disabled":true,"failed_active_vmms":0,"query_errors":[],"saga_errors":[],"sagas_completed":0,"sagas_started":0,"terminated_active_migrations":0}
106272025-04-30T06:43:08.526ZDEBGtest_snapshot_basic (BackgroundTasks): activating background_task = vpc_route_manager iteration = 2 reason = Timeout
106282025-04-30T06:43:08.526ZINFOtest_snapshot_basic (BackgroundTasks): VPC route manager running background_task = vpc_route_manager
106292025-04-30T06:43:08.526ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet
106302025-04-30T06:43:08.526ZTRACtest_snapshot_basic (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = vpc_route_manager resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
106312025-04-30T06:43:08.526ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database
106322025-04-30T06:43:08.526ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
106332025-04-30T06:43:08.526ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
106342025-04-30T06:43:08.527ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
106352025-04-30T06:43:08.537ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Inventory result = Ok(())
106362025-04-30T06:43:08.537ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database
106372025-04-30T06:43:08.537ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
106382025-04-30T06:43:08.540ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
106392025-04-30T06:43:08.542ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
106402025-04-30T06:43:08.547ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Fleet result = Ok(())
106412025-04-30T06:43:08.547ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database
106422025-04-30T06:43:08.547ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
106432025-04-30T06:43:08.548ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
106442025-04-30T06:43:08.548ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
106452025-04-30T06:43:08.554ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
106462025-04-30T06:43:08.554ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
106472025-04-30T06:43:08.554ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
106482025-04-30T06:43:08.555ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
106492025-04-30T06:43:08.555ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
106502025-04-30T06:43:08.555ZTRACtest_snapshot_basic (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
106512025-04-30T06:43:08.555ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
106522025-04-30T06:43:08.556ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {} }
106532025-04-30T06:43:08.556ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
106542025-04-30T06:43:08.557ZTRACtest_snapshot_basic (dns): buffer buffer = Length: 72 (0x48) bytes\n0000: 4f e6 01 00 00 01 00 00 00 00 00 01 09 5f 64 65 O............_de\n0010: 6e 64 72 69 74 65 04 5f 74 63 70 0d 63 6f 6e 74 ndrite._tcp.cont\n0020: 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 rol-plane.oxide.\n0030: 69 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 internal..!....)\n0040: 04 d0 00 00 00 00 00 00 ........ kind = dns peer_addr = [::1]:57762 req_id = 274d4489-ce89-4090-bfdd-93b3f6ea0f3b
106552025-04-30T06:43:08.557ZDEBGtest_snapshot_basic (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 20454,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_dendrite._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_dendrite._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 9,\n 95,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:57762 req_id = 274d4489-ce89-4090-bfdd-93b3f6ea0f3b
106562025-04-30T06:43:08.558ZDEBGtest_snapshot_basic (store): query key key = _dendrite._tcp
106572025-04-30T06:43:08.558ZDEBGtest_snapshot_basic (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
106582025-04-30T06:43:08.558ZDEBGtest_snapshot_basic (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:57762 query = LowerQuery { name: LowerName(Name("_dendrite._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 32976, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = 274d4489-ce89-4090-bfdd-93b3f6ea0f3b
106592025-04-30T06:43:08.558ZTRACtest_snapshot_basic (dns): buffer buffer = Length: 72 (0x48) bytes\n0000: b3 68 01 00 00 01 00 00 00 00 00 01 09 5f 64 65 .h..........._de\n0010: 6e 64 72 69 74 65 04 5f 74 63 70 0d 63 6f 6e 74 ndrite._tcp.cont\n0020: 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 rol-plane.oxide.\n0030: 69 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 internal..!....)\n0040: 04 d0 00 00 00 00 00 00 ........ kind = dns peer_addr = [::1]:61584 req_id = e052a2bd-12dc-474e-9250-2c1f3844404a
106602025-04-30T06:43:08.558ZDEBGtest_snapshot_basic (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 45928,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_dendrite._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_dendrite._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 9,\n 95,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:61584 req_id = e052a2bd-12dc-474e-9250-2c1f3844404a
106612025-04-30T06:43:08.559ZDEBGtest_snapshot_basic (store): query key key = _dendrite._tcp
106622025-04-30T06:43:08.559ZDEBGtest_snapshot_basic (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
106632025-04-30T06:43:08.559ZDEBGtest_snapshot_basic (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:61584 query = LowerQuery { name: LowerName(Name("_dendrite._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 32976, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = e052a2bd-12dc-474e-9250-2c1f3844404a
106642025-04-30T06:43:08.560ZDEBGtest_snapshot_basic (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 32976, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 4253, tv_nsec: 359578462 } })
106652025-04-30T06:43:08.560ZTRACtest_snapshot_basic (DnsResolver): lookup_service_targets: looking up SRV target
106662025-04-30T06:43:08.561ZDEBGtest_snapshot_basic (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 32976, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 4253, tv_nsec: 359937107 } })
106672025-04-30T06:43:08.561ZTRACtest_snapshot_basic (DnsResolver): lookup_service_targets: looking up SRV target
106682025-04-30T06:43:08.564ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
106692025-04-30T06:43:08.568ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Fleet result = Ok(())
106702025-04-30T06:43:08.568ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database
106712025-04-30T06:43:08.569ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
106722025-04-30T06:43:08.569ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
106732025-04-30T06:43:08.570ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
106742025-04-30T06:43:08.574ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
106752025-04-30T06:43:08.574ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet
106762025-04-30T06:43:08.574ZTRACtest_snapshot_basic (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = vpc_route_manager resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
106772025-04-30T06:43:08.576ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database
106782025-04-30T06:43:08.576ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
106792025-04-30T06:43:08.576ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
106802025-04-30T06:43:08.578ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database
106812025-04-30T06:43:08.578ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
106822025-04-30T06:43:08.579ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
106832025-04-30T06:43:08.579ZTRACtest_snapshot_basic (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal
106842025-04-30T06:43:08.595ZTRACtest_snapshot_basic (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: 8c 3b 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e .;..........-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:57266 req_id = 4ccbc20b-6ec0-4f8b-802a-bf216e3b28ed
106852025-04-30T06:43:08.596ZDEBGtest_snapshot_basic (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 35899,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:57266 req_id = 4ccbc20b-6ec0-4f8b-802a-bf216e3b28ed
106862025-04-30T06:43:08.596ZDEBGtest_snapshot_basic (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
106872025-04-30T06:43:08.596ZDEBGtest_snapshot_basic (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:57266 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = 4ccbc20b-6ec0-4f8b-802a-bf216e3b28ed
106882025-04-30T06:43:08.597ZTRACtest_snapshot_basic (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: 1d 78 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e .x..........-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:51571 req_id = b321890e-31ec-4313-9025-3ad436ca61b9
106892025-04-30T06:43:08.597ZDEBGtest_snapshot_basic (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 7544,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:51571 req_id = b321890e-31ec-4313-9025-3ad436ca61b9
106902025-04-30T06:43:08.597ZDEBGtest_snapshot_basic (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
106912025-04-30T06:43:08.598ZDEBGtest_snapshot_basic (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:51571 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = b321890e-31ec-4313-9025-3ad436ca61b9
106922025-04-30T06:43:08.598ZTRACtest_snapshot_basic (dns): buffer buffer = Length: 72 (0x48) bytes\n0000: 7e 11 01 00 00 01 00 00 00 00 00 01 09 5f 64 65 ~............_de\n0010: 6e 64 72 69 74 65 04 5f 74 63 70 0d 63 6f 6e 74 ndrite._tcp.cont\n0020: 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 rol-plane.oxide.\n0030: 69 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 internal..!....)\n0040: 04 d0 00 00 00 00 00 00 ........ kind = dns peer_addr = [::1]:58737 req_id = c33cc36f-65ec-4bcb-bcef-bf5a03983ee7
106932025-04-30T06:43:08.599ZDEBGtest_snapshot_basic (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 32273,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_dendrite._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_dendrite._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 9,\n 95,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:58737 req_id = c33cc36f-65ec-4bcb-bcef-bf5a03983ee7
106942025-04-30T06:43:08.599ZDEBGtest_snapshot_basic (store): query key key = _dendrite._tcp
106952025-04-30T06:43:08.600ZDEBGtest_snapshot_basic (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
106962025-04-30T06:43:08.600ZDEBGtest_snapshot_basic (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:58737 query = LowerQuery { name: LowerName(Name("_dendrite._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 32976, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = c33cc36f-65ec-4bcb-bcef-bf5a03983ee7
106972025-04-30T06:43:08.603ZINFOtest_snapshot_basic (BackgroundTasks): Determining switch slots managed by switch zones background_task = nat_v4_garbage_collector
106982025-04-30T06:43:08.660ZINFOtest_snapshot_basic (BackgroundTasks): determining switch slot managed by dendrite zone background_task = nat_v4_garbage_collector zone_address = ::1
106992025-04-30T06:43:08.660ZDEBGtest_snapshot_basic (MgsClient): client request background_task = nat_v4_garbage_collector body = None method = GET uri = http://[::1]:12225/local/switch-id
107002025-04-30T06:43:08.660ZDEBGtest_snapshot_basic (MgsClient): client response background_task = nat_v4_garbage_collector result = Err(reqwest::Error { kind: Request, url: "http://[::1]:12225/local/switch-id", source: hyper_util::client::legacy::Error(Connect, ConnectError("tcp connect error", Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) })
107012025-04-30T06:43:08.660ZWARNtest_snapshot_basic (BackgroundTasks): failed to identify switch slot for dendrite background_task = nat_v4_garbage_collector reason = Communication Error: error sending request for url (http://[::1]:12225/local/switch-id): client error (Connect): tcp connect error: Connection refused (os error 146): Connection refused (os error 146) zone_address = ::1
107022025-04-30T06:43:08.663ZWARNtest_snapshot_basic (BackgroundTasks): Failed to map switch zone addr: Communication Error: error sending request for url (http://[::1]:12225/local/switch-id), retrying background_task = nat_v4_garbage_collector
107032025-04-30T06:43:08.663ZINFOtest_snapshot_basic (BackgroundTasks): Determining switch slots managed by switch zones background_task = bfd_manager
107042025-04-30T06:43:08.710ZINFOtest_snapshot_basic (BackgroundTasks): determining switch slot managed by dendrite zone background_task = bfd_manager zone_address = ::1
107052025-04-30T06:43:08.710ZDEBGtest_snapshot_basic (MgsClient): client request background_task = bfd_manager body = None method = GET uri = http://[::1]:12225/local/switch-id
107062025-04-30T06:43:08.710ZDEBGtest_snapshot_basic (MgsClient): client response background_task = bfd_manager result = Err(reqwest::Error { kind: Request, url: "http://[::1]:12225/local/switch-id", source: hyper_util::client::legacy::Error(Connect, ConnectError("tcp connect error", Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) })
107072025-04-30T06:43:08.711ZWARNtest_snapshot_basic (BackgroundTasks): failed to identify switch slot for dendrite background_task = bfd_manager reason = Communication Error: error sending request for url (http://[::1]:12225/local/switch-id): client error (Connect): tcp connect error: Connection refused (os error 146): Connection refused (os error 146) zone_address = ::1
107082025-04-30T06:43:08.713ZWARNtest_snapshot_basic (BackgroundTasks): Failed to map switch zone addr: Communication Error: error sending request for url (http://[::1]:12225/local/switch-id), retrying background_task = bfd_manager
107092025-04-30T06:43:08.713ZDEBGtest_snapshot_basic (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 32976, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 4253, tv_nsec: 512707231 } })
107102025-04-30T06:43:08.713ZTRACtest_snapshot_basic (DnsResolver): lookup_service_targets: looking up SRV target
107112025-04-30T06:43:08.715ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database
107122025-04-30T06:43:08.715ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
107132025-04-30T06:43:08.718ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
107142025-04-30T06:43:08.719ZINFOtest_snapshot_basic (BackgroundTasks): all instance checks complete background_task = instance_watcher pruned_instances = 0 total_completed = 0 total_failed = 0 total_incomplete = 0 total_instances = 0 update_sagas_queued = 0
107152025-04-30T06:43:08.719ZDEBGtest_snapshot_basic (BackgroundTasks): activation complete background_task = instance_watcher elapsed = 195.138331ms iteration = 2 status = {"failed_checks":{},"incomplete_checks":{},"instance_states":{},"pruned_instances":0,"total_instances":0,"update_sagas_queued":0}
107162025-04-30T06:43:08.726ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
107172025-04-30T06:43:08.733ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
107182025-04-30T06:43:08.733ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database
107192025-04-30T06:43:08.733ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
107202025-04-30T06:43:08.734ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
107212025-04-30T06:43:08.737ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
107222025-04-30T06:43:08.756ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
107232025-04-30T06:43:08.756ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
107242025-04-30T06:43:08.756ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {} }
107252025-04-30T06:43:08.758ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
107262025-04-30T06:43:08.774ZTRACtest_snapshot_basic (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: 75 c4 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e u...........-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:64220 req_id = 8c6139ce-81a9-4b49-8a25-85e3449cfe9c
107272025-04-30T06:43:08.774ZDEBGtest_snapshot_basic (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 30148,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:64220 req_id = 8c6139ce-81a9-4b49-8a25-85e3449cfe9c
107282025-04-30T06:43:08.775ZDEBGtest_snapshot_basic (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
107292025-04-30T06:43:08.775ZDEBGtest_snapshot_basic (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:64220 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = 8c6139ce-81a9-4b49-8a25-85e3449cfe9c
107302025-04-30T06:43:08.776ZINFOtest_snapshot_basic (BackgroundTasks): Determining switch slots managed by switch zones background_task = switch_port_config_manager rack_id = c19a698f-c6f9-4a17-ae30-20d711b8f7dc
107312025-04-30T06:43:08.853ZINFOtest_snapshot_basic (BackgroundTasks): determining switch slot managed by dendrite zone background_task = switch_port_config_manager rack_id = c19a698f-c6f9-4a17-ae30-20d711b8f7dc zone_address = ::1
107322025-04-30T06:43:08.853ZDEBGtest_snapshot_basic (MgsClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = c19a698f-c6f9-4a17-ae30-20d711b8f7dc uri = http://[::1]:12225/local/switch-id
107332025-04-30T06:43:08.854ZDEBGtest_snapshot_basic (MgsClient): client response background_task = switch_port_config_manager rack_id = c19a698f-c6f9-4a17-ae30-20d711b8f7dc result = Err(reqwest::Error { kind: Request, url: "http://[::1]:12225/local/switch-id", source: hyper_util::client::legacy::Error(Connect, ConnectError("tcp connect error", Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) })
107342025-04-30T06:43:08.854ZWARNtest_snapshot_basic (BackgroundTasks): failed to identify switch slot for dendrite background_task = switch_port_config_manager rack_id = c19a698f-c6f9-4a17-ae30-20d711b8f7dc reason = Communication Error: error sending request for url (http://[::1]:12225/local/switch-id): client error (Connect): tcp connect error: Connection refused (os error 146): Connection refused (os error 146) zone_address = ::1
107352025-04-30T06:43:08.855ZWARNtest_snapshot_basic (BackgroundTasks): Failed to map switch zone addr: Communication Error: error sending request for url (http://[::1]:12225/local/switch-id), retrying background_task = switch_port_config_manager rack_id = c19a698f-c6f9-4a17-ae30-20d711b8f7dc
107362025-04-30T06:43:08.858ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet
107372025-04-30T06:43:08.859ZTRACtest_snapshot_basic (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = v2p_manager resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
107382025-04-30T06:43:08.859ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database
107392025-04-30T06:43:08.859ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
107402025-04-30T06:43:08.859ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
107412025-04-30T06:43:08.947ZINFOtest_snapshot_basic (BackgroundTasks): VPC route manager sled b6d65341-167c-41df-9b5c-41cded99c229 background_task = vpc_route_manager
107422025-04-30T06:43:08.947ZDEBGtest_snapshot_basic (BackgroundTasks): client request SledAgent = b6d65341-167c-41df-9b5c-41cded99c229 background_task = vpc_route_manager body = None method = GET uri = http://[::1]:41923/vpc-routes
107432025-04-30T06:43:08.951ZINFOtest_snapshot_basic (dropshot): accepted connection local_addr = [::1]:41923 remote_addr = [::1]:59813 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
107442025-04-30T06:43:08.951ZDEBGtest_snapshot_basic (collection-task): sent timer-based collection request to the collection task collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
107452025-04-30T06:43:08.951ZDEBGtest_snapshot_basic (collection-task): collecting from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
107462025-04-30T06:43:08.952ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
107472025-04-30T06:43:08.959ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
107482025-04-30T06:43:08.959ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet
107492025-04-30T06:43:08.959ZTRACtest_snapshot_basic (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = v2p_manager resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
107502025-04-30T06:43:08.959ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database
107512025-04-30T06:43:08.959ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
107522025-04-30T06:43:08.960ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
107532025-04-30T06:43:08.963ZTRACtest_snapshot_basic (dropshot): incoming request local_addr = [::1]:41923 method = GET remote_addr = [::1]:59813 req_id = 31a601c6-6202-4d06-801b-e77aa721ec5b sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /vpc-routes
107542025-04-30T06:43:08.966ZINFOtest_snapshot_basic (dropshot): request completed latency_us = 3476 local_addr = [::1]:41923 method = GET remote_addr = [::1]:59813 req_id = 31a601c6-6202-4d06-801b-e77aa721ec5b response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /vpc-routes
107552025-04-30T06:43:08.968ZDEBGtest_snapshot_basic (BackgroundTasks): client response SledAgent = b6d65341-167c-41df-9b5c-41cded99c229 background_task = vpc_route_manager result = Ok(Response { url: "http://[::1]:41923/vpc-routes", status: 200, headers: {"content-type": "application/json", "x-request-id": "31a601c6-6202-4d06-801b-e77aa721ec5b", "content-length": "2", "date": "Wed, 30 Apr 2025 06:43:08 GMT"} })
107562025-04-30T06:43:08.968ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database
107572025-04-30T06:43:08.968ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
107582025-04-30T06:43:08.968ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
107592025-04-30T06:43:08.968ZTRACtest_snapshot_basic (collection-task): sent collection request to producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
107602025-04-30T06:43:08.971ZDEBGtest_snapshot_basic (collection-task): collected results from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 n_results = 1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
107612025-04-30T06:43:08.971ZTRACtest_snapshot_basic (collection-task): forwarded results to main collection loop collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
107622025-04-30T06:43:08.971ZTRACtest_snapshot_basic (collection-task): top of inner collection loop, waiting for next request collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
107632025-04-30T06:43:08.971ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
107642025-04-30T06:43:08.993ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
107652025-04-30T06:43:08.994ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database
107662025-04-30T06:43:08.994ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
107672025-04-30T06:43:08.994ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
107682025-04-30T06:43:09.063ZDEBGtest_snapshot_basic (BackgroundTasks): client request SledAgent = b6d65341-167c-41df-9b5c-41cded99c229 background_task = v2p_manager body = None method = GET uri = http://[::1]:41923/v2p
107692025-04-30T06:43:09.068ZINFOtest_snapshot_basic (dropshot): accepted connection local_addr = [::1]:41923 remote_addr = [::1]:56071 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
107702025-04-30T06:43:09.068ZDEBGtest_snapshot_basic (results-sink): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
107712025-04-30T06:43:09.068ZDEBGtest_snapshot_basic (results-sink-cluster): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
107722025-04-30T06:43:09.069ZDEBGtest_snapshot_basic (clickhouse-client): unrolling 1 total samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 6ad06c6b-9a6e-4ed1-b3f4-5d2e9da3f8a7
107732025-04-30T06:43:09.071ZTRACtest_snapshot_basic (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 6ad06c6b-9a6e-4ed1-b3f4-5d2e9da3f8a7 n_columns = 4 n_rows = 2 sql = INSERT INTO oximeter.fields_string FORMAT Native
107742025-04-30T06:43:09.072ZTRACtest_snapshot_basic (dropshot): incoming request local_addr = [::1]:41923 method = GET remote_addr = [::1]:56071 req_id = 360677fe-6807-469b-aebe-42379a59eafa sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /v2p
107752025-04-30T06:43:09.104ZINFOtest_snapshot_basic (dropshot): request completed latency_us = 32126 local_addr = [::1]:41923 method = GET remote_addr = [::1]:56071 req_id = 360677fe-6807-469b-aebe-42379a59eafa response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /v2p
107762025-04-30T06:43:09.106ZDEBGtest_snapshot_basic (BackgroundTasks): client response SledAgent = b6d65341-167c-41df-9b5c-41cded99c229 background_task = v2p_manager result = Ok(Response { url: "http://[::1]:41923/v2p", status: 200, headers: {"content-type": "application/json", "x-request-id": "360677fe-6807-469b-aebe-42379a59eafa", "content-length": "2", "date": "Wed, 30 Apr 2025 06:43:08 GMT"} })
107772025-04-30T06:43:09.106ZINFOtest_snapshot_basic (BackgroundTasks): found opte v2p mappings background_task = v2p_manager interfaces = {} sled = sim-b6d65341
107782025-04-30T06:43:09.106ZINFOtest_snapshot_basic (BackgroundTasks): v2p mappings to delete background_task = v2p_manager mappings = [] sled = sim-b6d65341
107792025-04-30T06:43:09.106ZINFOtest_snapshot_basic (BackgroundTasks): v2p mappings to add background_task = v2p_manager mappings = [] sled = sim-b6d65341
107802025-04-30T06:43:09.108ZDEBGtest_snapshot_basic (BackgroundTasks): activation complete background_task = v2p_manager elapsed = 585.476152ms iteration = 2 status = {}
107812025-04-30T06:43:09.109ZDEBGtest_snapshot_basic (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 6ad06c6b-9a6e-4ed1-b3f4-5d2e9da3f8a7 n_rows = 2 table_name = fields_string
107822025-04-30T06:43:09.109ZTRACtest_snapshot_basic (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 6ad06c6b-9a6e-4ed1-b3f4-5d2e9da3f8a7 n_columns = 4 n_rows = 1 sql = INSERT INTO oximeter.measurements_i64 FORMAT Native
107832025-04-30T06:43:09.115ZDEBGtest_snapshot_basic (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 6ad06c6b-9a6e-4ed1-b3f4-5d2e9da3f8a7 n_rows = 1 table_name = measurements_i64
107842025-04-30T06:43:09.115ZTRACtest_snapshot_basic (results-sink): successfully inserted samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
107852025-04-30T06:43:09.128ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database
107862025-04-30T06:43:09.128ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
107872025-04-30T06:43:09.129ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
107882025-04-30T06:43:09.129ZWARNtest_snapshot_basic (datastore): Non-retryable transaction failure err = DatabaseError(UniqueViolation, "duplicate key value violates unique constraint \\"inv_caboose_pkey\\"") transaction = inventory_insert_collection
107892025-04-30T06:43:09.129ZWARNtest_snapshot_basic (BackgroundTasks): inventory collection failed background_task = inventory_collection error = failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint "inv_caboose_pkey"
107902025-04-30T06:43:09.129ZDEBGtest_snapshot_basic (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 997.455826ms iteration = 37 status = {"error":"failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint \\"inv_caboose_pkey\\""}
107912025-04-30T06:43:09.129ZDEBGtest_snapshot_basic (BackgroundTasks): activating background_task = inventory_collection iteration = 38 reason = Signaled
107922025-04-30T06:43:09.130ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
107932025-04-30T06:43:09.130ZTRACtest_snapshot_basic (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
107942025-04-30T06:43:09.130ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
107952025-04-30T06:43:09.130ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
107962025-04-30T06:43:09.131ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
107972025-04-30T06:43:09.131ZINFOtest_snapshot_basic (BackgroundTasks): computed internet gateway mappings for sled assocs = {} background_task = vpc_route_manager sled = sim-b6d65341
107982025-04-30T06:43:09.131ZDEBGtest_snapshot_basic (BackgroundTasks): client request SledAgent = b6d65341-167c-41df-9b5c-41cded99c229 background_task = vpc_route_manager body = Some(Body) method = PUT uri = http://[::1]:41923/eip-gateways
107992025-04-30T06:43:09.136ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
108002025-04-30T06:43:09.136ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {} }
108012025-04-30T06:43:09.137ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
108022025-04-30T06:43:09.137ZTRACtest_snapshot_basic (dropshot): incoming request local_addr = [::1]:41923 method = PUT remote_addr = [::1]:59813 req_id = 1df20e1d-af49-4dc6-8274-eb0d78b61568 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /eip-gateways
108032025-04-30T06:43:09.138ZINFOtest_snapshot_basic (dropshot): request completed latency_us = 1008 local_addr = [::1]:41923 method = PUT remote_addr = [::1]:59813 req_id = 1df20e1d-af49-4dc6-8274-eb0d78b61568 response_code = 204 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /eip-gateways
108042025-04-30T06:43:09.138ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Sled { parent: Fleet, key: b6d65341-167c-41df-9b5c-41cded99c229, lookup_type: ById(b6d65341-167c-41df-9b5c-41cded99c229) }
108052025-04-30T06:43:09.138ZTRACtest_snapshot_basic (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = service_zone_nat_tracker resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
108062025-04-30T06:43:09.138ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database
108072025-04-30T06:43:09.138ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
108082025-04-30T06:43:09.139ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
108092025-04-30T06:43:09.139ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
108102025-04-30T06:43:09.144ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
108112025-04-30T06:43:09.144ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
108122025-04-30T06:43:09.144ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
108132025-04-30T06:43:09.144ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
108142025-04-30T06:43:09.145ZDEBGtest_snapshot_basic (BackgroundTasks): client response SledAgent = b6d65341-167c-41df-9b5c-41cded99c229 background_task = vpc_route_manager result = Ok(Response { url: "http://[::1]:41923/eip-gateways", status: 204, headers: {"x-request-id": "1df20e1d-af49-4dc6-8274-eb0d78b61568", "date": "Wed, 30 Apr 2025 06:43:08 GMT"} })
108152025-04-30T06:43:09.148ZDEBGtest_snapshot_basic (BackgroundTasks): activation complete background_task = vpc_route_manager elapsed = 622.063631ms iteration = 2 status = {}
108162025-04-30T06:43:09.150ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
108172025-04-30T06:43:09.155ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Sled { parent: Fleet, key: b6d65341-167c-41df-9b5c-41cded99c229, lookup_type: ById(b6d65341-167c-41df-9b5c-41cded99c229) } result = Ok(())
108182025-04-30T06:43:09.155ZERROtest_snapshot_basic (BackgroundTasks): ipv6 addresses for service zone nat not implemented background_task = service_zone_nat_tracker
108192025-04-30T06:43:09.156ZERROtest_snapshot_basic (BackgroundTasks): generated config for fewer than the minimum allowed number of dns zones background_task = service_zone_nat_tracker
108202025-04-30T06:43:09.156ZDEBGtest_snapshot_basic (BackgroundTasks): activation complete background_task = service_zone_nat_tracker elapsed = 687.127487ms iteration = 2 status = {"error":"generated config for fewer than the minimum allowed number of dns zones"}
108212025-04-30T06:43:09.156ZDEBGtest_snapshot_basic (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = [(d1ad44e5-18dd-4193-a70e-bf9042f0e0e1, 0)]
108222025-04-30T06:43:09.156ZTRACtest_snapshot_basic (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
108232025-04-30T06:43:09.158ZTRACtest_snapshot_basic (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: 9e 2f 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 ./..........._mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:54848 req_id = 19d4fd11-7c0b-4e6d-9a9b-7b6fa79812d8
108242025-04-30T06:43:09.158ZDEBGtest_snapshot_basic (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 40495,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:54848 req_id = 19d4fd11-7c0b-4e6d-9a9b-7b6fa79812d8
108252025-04-30T06:43:09.158ZDEBGtest_snapshot_basic (store): query key key = _mgs._tcp
108262025-04-30T06:43:09.159ZDEBGtest_snapshot_basic (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
108272025-04-30T06:43:09.159ZDEBGtest_snapshot_basic (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:54848 query = LowerQuery { name: LowerName(Name("_mgs._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 41502, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = 19d4fd11-7c0b-4e6d-9a9b-7b6fa79812d8
108282025-04-30T06:43:09.159ZDEBGtest_snapshot_basic (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 41502, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 4253, tv_nsec: 958606135 } })
108292025-04-30T06:43:09.159ZTRACtest_snapshot_basic (DnsResolver): lookup_service_targets: looking up SRV target
108302025-04-30T06:43:09.160ZTRACtest_snapshot_basic (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: b2 4b 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e .K..........-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:52109 req_id = b833573c-46c8-4ecf-b789-233a6c13a00a
108312025-04-30T06:43:09.160ZDEBGtest_snapshot_basic (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 45643,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:52109 req_id = b833573c-46c8-4ecf-b789-233a6c13a00a
108322025-04-30T06:43:09.160ZDEBGtest_snapshot_basic (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
108332025-04-30T06:43:09.160ZDEBGtest_snapshot_basic (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:52109 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = b833573c-46c8-4ecf-b789-233a6c13a00a
108342025-04-30T06:43:09.220ZTRACtest_snapshot_basic (DnsResolver): lookup_all_socket_v6 srv dns_name = _clickhouse-admin-keeper._tcp.control-plane.oxide.internal
108352025-04-30T06:43:09.221ZWARNtest_snapshot_basic (results-sink-cluster): failed to insert some results into metric DB collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 error = Failed to check out connection to database: No backends found for this service file = oximeter/collector/src/results_sink.rs:92
108362025-04-30T06:43:09.221ZTRACtest_snapshot_basic (dns): buffer buffer = Length: 87 (0x57) bytes\n0000: 9f 7d 01 00 00 01 00 00 00 00 00 01 18 5f 63 6c .}..........._cl\n0010: 69 63 6b 68 6f 75 73 65 2d 61 64 6d 69 6e 2d 6b ickhouse-admin-k\n0020: 65 65 70 65 72 04 5f 74 63 70 0d 63 6f 6e 74 72 eeper._tcp.contr\n0030: 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 69 ol-plane.oxide.i\n0040: 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 04 nternal..!....).\n0050: d0 00 00 00 00 00 00 ....... kind = dns peer_addr = [::1]:58454 req_id = 18362f7b-eac5-4299-a17f-8e7ba732ce6a
108372025-04-30T06:43:09.221ZDEBGtest_snapshot_basic (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 40829,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 24,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 97,\n 100,\n 109,\n 105,\n 110,\n 45,\n 107,\n 101,\n 101,\n 112,\n 101,\n 114,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:58454 req_id = 18362f7b-eac5-4299-a17f-8e7ba732ce6a
108382025-04-30T06:43:09.221ZDEBGtest_snapshot_basic (store): query key key = _clickhouse-admin-keeper._tcp
108392025-04-30T06:43:09.221ZERROtest_snapshot_basic (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 40829, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } NXDOMAIN: no records found for name: "_clickhouse-admin-keeper._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:58454 req_id = 18362f7b-eac5-4299-a17f-8e7ba732ce6a
108402025-04-30T06:43:09.221ZDEBGtest_snapshot_basic (BackgroundTasks): begin collection background_task = inventory_collection
108412025-04-30T06:43:09.222ZDEBGtest_snapshot_basic (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:41502
108422025-04-30T06:43:09.222ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/ignition
108432025-04-30T06:43:09.225ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "369c9f7c-9952-44da-832b-64ef88e0b12c", "content-length": "835", "date": "Wed, 30 Apr 2025 06:43:08 GMT"} })
108442025-04-30T06:43:09.225ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/0
108452025-04-30T06:43:09.228ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "1f5c3ea6-ad80-445e-8c88-628556695487", "content-length": "734", "date": "Wed, 30 Apr 2025 06:43:08 GMT"} })
108462025-04-30T06:43:09.231ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/0/component/sp/caboose?firmware_slot=0
108472025-04-30T06:43:09.234ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "eff87415-82ab-4cd8-9ec9-54fca683976f", "content-length": "111", "date": "Wed, 30 Apr 2025 06:43:08 GMT"} })
108482025-04-30T06:43:09.234ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/0/component/sp/caboose?firmware_slot=1
108492025-04-30T06:43:09.236ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "375777f2-1a11-4c08-b540-56114cc16a5e", "content-length": "111", "date": "Wed, 30 Apr 2025 06:43:08 GMT"} })
108502025-04-30T06:43:09.236ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/0/component/rot/caboose?firmware_slot=0
108512025-04-30T06:43:09.245ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "3f0a75ef-4cb6-483d-bdd1-0c5568ff8932", "content-length": "170", "date": "Wed, 30 Apr 2025 06:43:08 GMT"} })
108522025-04-30T06:43:09.245ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/0/component/rot/caboose?firmware_slot=1
108532025-04-30T06:43:09.249ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "1a4a3d67-8758-4d82-b522-6858f070d355", "content-length": "170", "date": "Wed, 30 Apr 2025 06:43:08 GMT"} })
108542025-04-30T06:43:09.249ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/0/component/stage0/caboose?firmware_slot=0
108552025-04-30T06:43:09.252ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "34b3f84f-5c5d-45f0-8daf-5ddf9d6fccf1", "content-length": "179", "date": "Wed, 30 Apr 2025 06:43:08 GMT"} })
108562025-04-30T06:43:09.252ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/0/component/stage0/caboose?firmware_slot=1
108572025-04-30T06:43:09.255ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "04a40c97-e523-4370-858c-912ef5130a86", "content-length": "179", "date": "Wed, 30 Apr 2025 06:43:08 GMT"} })
108582025-04-30T06:43:09.255ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/0/component/rot/cmpa
108592025-04-30T06:43:09.257ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "67ca9bb9-3747-4cd5-a016-f69dc69e59d8", "content-length": "702", "date": "Wed, 30 Apr 2025 06:43:08 GMT"} })
108602025-04-30T06:43:09.257ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/0/component/rot/cfpa
108612025-04-30T06:43:09.259ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "b3a544db-3adf-4938-9ed5-61a45fc4830c", "content-length": "727", "date": "Wed, 30 Apr 2025 06:43:08 GMT"} })
108622025-04-30T06:43:09.259ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/0/component/rot/cfpa
108632025-04-30T06:43:09.260ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "189a1f5a-1d8b-45d3-8464-bf225a05ecb9", "content-length": "729", "date": "Wed, 30 Apr 2025 06:43:08 GMT"} })
108642025-04-30T06:43:09.261ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/0/component/rot/cfpa
108652025-04-30T06:43:09.262ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "82966d77-92f5-4537-a9a8-509945b72ef5", "content-length": "728", "date": "Wed, 30 Apr 2025 06:43:08 GMT"} })
108662025-04-30T06:43:09.262ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/1
108672025-04-30T06:43:09.282ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "3f343ddf-2951-475b-8737-cf3894d37f60", "content-length": "734", "date": "Wed, 30 Apr 2025 06:43:08 GMT"} })
108682025-04-30T06:43:09.282ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/1/component/sp/caboose?firmware_slot=0
108692025-04-30T06:43:09.291ZDEBGtest_snapshot_basic: attempt to fetch latest inventory collection result = Ok("found")
108702025-04-30T06:43:09.292ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "ecb25156-5765-4799-b322-e2982b28300e", "content-length": "111", "date": "Wed, 30 Apr 2025 06:43:08 GMT"} })
108712025-04-30T06:43:09.292ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/1/component/sp/caboose?firmware_slot=1
108722025-04-30T06:43:09.298ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "fcc99f8f-0cbd-4cb6-b213-709745c8bef7", "content-length": "111", "date": "Wed, 30 Apr 2025 06:43:08 GMT"} })
108732025-04-30T06:43:09.298ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/1/component/rot/caboose?firmware_slot=0
108742025-04-30T06:43:09.300ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "a6acff53-16e5-4024-996f-53558a1d3a2b", "content-length": "170", "date": "Wed, 30 Apr 2025 06:43:08 GMT"} })
108752025-04-30T06:43:09.300ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/1/component/rot/caboose?firmware_slot=1
108762025-04-30T06:43:09.303ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "08967ec3-86e0-48ef-bff9-e9d016fbe7d8", "content-length": "170", "date": "Wed, 30 Apr 2025 06:43:08 GMT"} })
108772025-04-30T06:43:09.303ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/1/component/stage0/caboose?firmware_slot=0
108782025-04-30T06:43:09.306ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "8f61ba01-912a-405e-8b00-c7f61e926e4b", "content-length": "179", "date": "Wed, 30 Apr 2025 06:43:08 GMT"} })
108792025-04-30T06:43:09.306ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/1/component/stage0/caboose?firmware_slot=1
108802025-04-30T06:43:09.311ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "796d2de1-427d-482c-b95a-2e5224994031", "content-length": "179", "date": "Wed, 30 Apr 2025 06:43:08 GMT"} })
108812025-04-30T06:43:09.311ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/1/component/rot/cmpa
108822025-04-30T06:43:09.313ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "7f152443-11b8-40a8-9fe6-4ef4999a6ebc", "content-length": "702", "date": "Wed, 30 Apr 2025 06:43:08 GMT"} })
108832025-04-30T06:43:09.313ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/1/component/rot/cfpa
108842025-04-30T06:43:09.314ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "49e4cd0e-6222-4f31-898f-33a883161e9c", "content-length": "727", "date": "Wed, 30 Apr 2025 06:43:08 GMT"} })
108852025-04-30T06:43:09.314ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/1/component/rot/cfpa
108862025-04-30T06:43:09.315ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "a80b9dc7-9103-40a0-8094-ff929ef2954b", "content-length": "729", "date": "Wed, 30 Apr 2025 06:43:08 GMT"} })
108872025-04-30T06:43:09.315ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/1/component/rot/cfpa
108882025-04-30T06:43:09.317ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "275df4e2-3463-42f0-9567-89e191aa3a5e", "content-length": "728", "date": "Wed, 30 Apr 2025 06:43:08 GMT"} })
108892025-04-30T06:43:09.318ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/0
108902025-04-30T06:43:09.319ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "29bf40a2-d764-44e3-9086-016947701c48", "content-length": "723", "date": "Wed, 30 Apr 2025 06:43:08 GMT"} })
108912025-04-30T06:43:09.319ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/0/component/sp/caboose?firmware_slot=0
108922025-04-30T06:43:09.322ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "f71024ff-779f-45db-846c-ff88102e1c66", "content-length": "109", "date": "Wed, 30 Apr 2025 06:43:08 GMT"} })
108932025-04-30T06:43:09.323ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/0/component/sp/caboose?firmware_slot=1
108942025-04-30T06:43:09.325ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "58c5607d-a85e-4704-bfb9-b68c012847f9", "content-length": "109", "date": "Wed, 30 Apr 2025 06:43:08 GMT"} })
108952025-04-30T06:43:09.326ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/0/component/rot/caboose?firmware_slot=0
108962025-04-30T06:43:09.327ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "f9f47450-602d-41b7-97c2-48c6c848ae3a", "content-length": "169", "date": "Wed, 30 Apr 2025 06:43:08 GMT"} })
108972025-04-30T06:43:09.327ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/0/component/rot/caboose?firmware_slot=1
108982025-04-30T06:43:09.330ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "b9855c74-5869-4a00-8a5a-4e31f08138bf", "content-length": "169", "date": "Wed, 30 Apr 2025 06:43:08 GMT"} })
108992025-04-30T06:43:09.330ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/0/component/stage0/caboose?firmware_slot=0
109002025-04-30T06:43:09.332ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "07fb5a7f-ea81-4f0a-82c2-97f830125841", "content-length": "178", "date": "Wed, 30 Apr 2025 06:43:08 GMT"} })
109012025-04-30T06:43:09.332ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/0/component/stage0/caboose?firmware_slot=1
109022025-04-30T06:43:09.336ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "635ade2a-3673-4b27-a797-b5999c196816", "content-length": "178", "date": "Wed, 30 Apr 2025 06:43:08 GMT"} })
109032025-04-30T06:43:09.336ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/0/component/rot/cmpa
109042025-04-30T06:43:09.337ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "930660e7-f273-4ea8-be45-f9403f624403", "content-length": "702", "date": "Wed, 30 Apr 2025 06:43:08 GMT"} })
109052025-04-30T06:43:09.337ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/0/component/rot/cfpa
109062025-04-30T06:43:09.338ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "227e39fd-bd98-4981-9e2a-7566f346612c", "content-length": "727", "date": "Wed, 30 Apr 2025 06:43:08 GMT"} })
109072025-04-30T06:43:09.338ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/0/component/rot/cfpa
109082025-04-30T06:43:09.339ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "d8879ea1-bcfc-4f10-8a97-992a30897dba", "content-length": "729", "date": "Wed, 30 Apr 2025 06:43:08 GMT"} })
109092025-04-30T06:43:09.339ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/0/component/rot/cfpa
109102025-04-30T06:43:09.340ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "bbad2657-ca74-4641-963b-8fb28745a685", "content-length": "728", "date": "Wed, 30 Apr 2025 06:43:08 GMT"} })
109112025-04-30T06:43:09.340ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/1
109122025-04-30T06:43:09.341ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "84464a5b-002d-420c-a68b-0a1d477f3cd5", "content-length": "723", "date": "Wed, 30 Apr 2025 06:43:08 GMT"} })
109132025-04-30T06:43:09.341ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/1/component/sp/caboose?firmware_slot=0
109142025-04-30T06:43:09.342ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
109152025-04-30T06:43:09.342ZTRACtest_snapshot_basic (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
109162025-04-30T06:43:09.342ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
109172025-04-30T06:43:09.342ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {} }
109182025-04-30T06:43:09.342ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
109192025-04-30T06:43:09.344ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "f534238c-2189-4200-a5a9-a4a6c193bc26", "content-length": "109", "date": "Wed, 30 Apr 2025 06:43:08 GMT"} })
109202025-04-30T06:43:09.345ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/1/component/sp/caboose?firmware_slot=1
109212025-04-30T06:43:09.346ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "f5e26e6a-d694-43df-b62a-97b78b3c7334", "content-length": "109", "date": "Wed, 30 Apr 2025 06:43:08 GMT"} })
109222025-04-30T06:43:09.346ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/1/component/rot/caboose?firmware_slot=0
109232025-04-30T06:43:09.348ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
109242025-04-30T06:43:09.353ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
109252025-04-30T06:43:09.353ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
109262025-04-30T06:43:09.353ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {} }
109272025-04-30T06:43:09.354ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
109282025-04-30T06:43:09.354ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "bcb3fef5-e3f7-49fb-800f-721b93ccb26c", "content-length": "169", "date": "Wed, 30 Apr 2025 06:43:08 GMT"} })
109292025-04-30T06:43:09.354ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/1/component/rot/caboose?firmware_slot=1
109302025-04-30T06:43:09.356ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "3004a730-3bd8-4f40-a6e6-5150ba3fb30b", "content-length": "169", "date": "Wed, 30 Apr 2025 06:43:08 GMT"} })
109312025-04-30T06:43:09.356ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/1/component/stage0/caboose?firmware_slot=0
109322025-04-30T06:43:09.362ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "e3213df0-8574-453e-aa38-1d0dcba985ea", "content-length": "178", "date": "Wed, 30 Apr 2025 06:43:08 GMT"} })
109332025-04-30T06:43:09.362ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/1/component/stage0/caboose?firmware_slot=1
109342025-04-30T06:43:09.362ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
109352025-04-30T06:43:09.362ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {} }
109362025-04-30T06:43:09.363ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
109372025-04-30T06:43:09.365ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "eac440e2-08a3-4939-bcb5-97eecb90b631", "content-length": "178", "date": "Wed, 30 Apr 2025 06:43:08 GMT"} })
109382025-04-30T06:43:09.365ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/1/component/rot/cmpa
109392025-04-30T06:43:09.366ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "79e5580c-7fc7-483f-9aed-e32443c2f493", "content-length": "702", "date": "Wed, 30 Apr 2025 06:43:08 GMT"} })
109402025-04-30T06:43:09.366ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/1/component/rot/cfpa
109412025-04-30T06:43:09.367ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "8a2c8560-0b35-4721-ac3e-336bee3da8ac", "content-length": "727", "date": "Wed, 30 Apr 2025 06:43:08 GMT"} })
109422025-04-30T06:43:09.367ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/1/component/rot/cfpa
109432025-04-30T06:43:09.368ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "a930f56f-0291-419f-964a-8ceea705f4bc", "content-length": "729", "date": "Wed, 30 Apr 2025 06:43:08 GMT"} })
109442025-04-30T06:43:09.368ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/1/component/rot/cfpa
109452025-04-30T06:43:09.369ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "209bc497-5688-4e47-8192-733655e7c38f", "content-length": "728", "date": "Wed, 30 Apr 2025 06:43:08 GMT"} })
109462025-04-30T06:43:09.369ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
109472025-04-30T06:43:09.369ZTRACtest_snapshot_basic (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
109482025-04-30T06:43:09.369ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
109492025-04-30T06:43:09.369ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
109502025-04-30T06:43:09.370ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
109512025-04-30T06:43:09.377ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
109522025-04-30T06:43:09.381ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
109532025-04-30T06:43:09.381ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
109542025-04-30T06:43:09.381ZTRACtest_snapshot_basic (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
109552025-04-30T06:43:09.381ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
109562025-04-30T06:43:09.381ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
109572025-04-30T06:43:09.382ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
109582025-04-30T06:43:09.386ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
109592025-04-30T06:43:09.391ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
109602025-04-30T06:43:09.391ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
109612025-04-30T06:43:09.391ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
109622025-04-30T06:43:09.394ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
109632025-04-30T06:43:09.495ZDEBGtest_snapshot_basic (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:41923
109642025-04-30T06:43:09.496ZDEBGtest_snapshot_basic (BackgroundTasks): client request SledAgent = http://[::1]:41923 background_task = inventory_collection body = None method = GET uri = http://[::1]:41923/inventory
109652025-04-30T06:43:09.497ZINFOtest_snapshot_basic (dropshot): accepted connection local_addr = [::1]:41923 remote_addr = [::1]:35233 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
109662025-04-30T06:43:09.498ZTRACtest_snapshot_basic (dropshot): incoming request local_addr = [::1]:41923 method = GET remote_addr = [::1]:35233 req_id = a79434a2-6b16-45fb-933b-6be14c2748ae sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
109672025-04-30T06:43:09.499ZINFOtest_snapshot_basic (dropshot): request completed latency_us = 760 local_addr = [::1]:41923 method = GET remote_addr = [::1]:35233 req_id = a79434a2-6b16-45fb-933b-6be14c2748ae response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
109682025-04-30T06:43:09.499ZDEBGtest_snapshot_basic (BackgroundTasks): client response SledAgent = http://[::1]:41923 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:41923/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "a79434a2-6b16-45fb-933b-6be14c2748ae", "content-length": "3141", "date": "Wed, 30 Apr 2025 06:43:08 GMT"} })
109692025-04-30T06:43:09.502ZDEBGtest_snapshot_basic (BackgroundTasks): begin collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
109702025-04-30T06:43:09.502ZDEBGtest_snapshot_basic (BackgroundTasks): end collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
109712025-04-30T06:43:09.502ZDEBGtest_snapshot_basic (BackgroundTasks): finished collection background_task = inventory_collection
109722025-04-30T06:43:09.506ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
109732025-04-30T06:43:09.506ZTRACtest_snapshot_basic (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
109742025-04-30T06:43:09.507ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
109752025-04-30T06:43:09.507ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
109762025-04-30T06:43:09.509ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
109772025-04-30T06:43:09.517ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
109782025-04-30T06:43:09.522ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
109792025-04-30T06:43:09.522ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
109802025-04-30T06:43:09.522ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
109812025-04-30T06:43:09.526ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
109822025-04-30T06:43:09.571ZDEBGtest_snapshot_basic: attempt to fetch latest inventory collection result = Ok("found")
109832025-04-30T06:43:09.622ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
109842025-04-30T06:43:09.622ZTRACtest_snapshot_basic (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
109852025-04-30T06:43:09.622ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
109862025-04-30T06:43:09.622ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {} }
109872025-04-30T06:43:09.623ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
109882025-04-30T06:43:09.625ZWARNtest_snapshot_basic (datastore): Non-retryable transaction failure err = DatabaseError(UniqueViolation, "duplicate key value violates unique constraint \\"inv_caboose_pkey\\"") transaction = inventory_insert_collection
109892025-04-30T06:43:09.627ZWARNtest_snapshot_basic (BackgroundTasks): inventory collection failed background_task = inventory_collection error = failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint "inv_caboose_pkey"
109902025-04-30T06:43:09.627ZDEBGtest_snapshot_basic (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 497.643412ms iteration = 38 status = {"error":"failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint \\"inv_caboose_pkey\\""}
109912025-04-30T06:43:09.627ZDEBGtest_snapshot_basic (BackgroundTasks): activating background_task = inventory_collection iteration = 39 reason = Signaled
109922025-04-30T06:43:09.627ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
109932025-04-30T06:43:09.627ZTRACtest_snapshot_basic (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
109942025-04-30T06:43:09.627ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
109952025-04-30T06:43:09.628ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
109962025-04-30T06:43:09.628ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
109972025-04-30T06:43:09.632ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
109982025-04-30T06:43:09.646ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
109992025-04-30T06:43:09.646ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
110002025-04-30T06:43:09.646ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {} }
110012025-04-30T06:43:09.646ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
110022025-04-30T06:43:09.649ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
110032025-04-30T06:43:09.649ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {} }
110042025-04-30T06:43:09.650ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
110052025-04-30T06:43:09.650ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
110062025-04-30T06:43:09.656ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
110072025-04-30T06:43:09.656ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
110082025-04-30T06:43:09.656ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
110092025-04-30T06:43:09.657ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
110102025-04-30T06:43:09.662ZDEBGtest_snapshot_basic (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = [(d1ad44e5-18dd-4193-a70e-bf9042f0e0e1, 0)]
110112025-04-30T06:43:09.662ZTRACtest_snapshot_basic (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
110122025-04-30T06:43:09.663ZTRACtest_snapshot_basic (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: 0b 77 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 .w..........._mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:53556 req_id = d6cc92ac-4e5e-4b67-b4ed-ac2692e16461
110132025-04-30T06:43:09.663ZDEBGtest_snapshot_basic (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 2935,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:53556 req_id = d6cc92ac-4e5e-4b67-b4ed-ac2692e16461
110142025-04-30T06:43:09.663ZDEBGtest_snapshot_basic (store): query key key = _mgs._tcp
110152025-04-30T06:43:09.664ZDEBGtest_snapshot_basic (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
110162025-04-30T06:43:09.664ZDEBGtest_snapshot_basic (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:53556 query = LowerQuery { name: LowerName(Name("_mgs._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 41502, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = d6cc92ac-4e5e-4b67-b4ed-ac2692e16461
110172025-04-30T06:43:09.665ZDEBGtest_snapshot_basic (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 41502, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 4254, tv_nsec: 464551779 } })
110182025-04-30T06:43:09.665ZTRACtest_snapshot_basic (DnsResolver): lookup_service_targets: looking up SRV target
110192025-04-30T06:43:09.666ZTRACtest_snapshot_basic (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: d5 78 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e .x..........-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:49953 req_id = 821276b4-b86e-45ab-9fec-ac771c7e39a9
110202025-04-30T06:43:09.666ZDEBGtest_snapshot_basic (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 54648,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:49953 req_id = 821276b4-b86e-45ab-9fec-ac771c7e39a9
110212025-04-30T06:43:09.666ZDEBGtest_snapshot_basic (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
110222025-04-30T06:43:09.666ZDEBGtest_snapshot_basic (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:49953 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = 821276b4-b86e-45ab-9fec-ac771c7e39a9
110232025-04-30T06:43:09.734ZTRACtest_snapshot_basic (DnsResolver): lookup_all_socket_v6 srv dns_name = _clickhouse-admin-keeper._tcp.control-plane.oxide.internal
110242025-04-30T06:43:09.734ZINFOtest_snapshot_basic (oximeter-agent): refreshing list of producers from Nexus collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 file = oximeter/collector/src/agent.rs:562
110252025-04-30T06:43:09.736ZDEBGtest_snapshot_basic (oximeter): client request body = None method = GET uri = http://127.0.0.1:61468/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
110262025-04-30T06:43:09.736ZTRACtest_snapshot_basic (dropshot_internal): incoming request local_addr = 127.0.0.1:61468 method = GET remote_addr = 127.0.0.1:64332 req_id = 08d6672b-37b4-4e25-9771-f72334eddb1c uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
110272025-04-30T06:43:09.737ZTRACtest_snapshot_basic (dns): buffer buffer = Length: 87 (0x57) bytes\n0000: dd c9 01 00 00 01 00 00 00 00 00 01 18 5f 63 6c ............._cl\n0010: 69 63 6b 68 6f 75 73 65 2d 61 64 6d 69 6e 2d 6b ickhouse-admin-k\n0020: 65 65 70 65 72 04 5f 74 63 70 0d 63 6f 6e 74 72 eeper._tcp.contr\n0030: 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 69 ol-plane.oxide.i\n0040: 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 04 nternal..!....).\n0050: d0 00 00 00 00 00 00 ....... kind = dns peer_addr = [::1]:63319 req_id = 86688c77-5071-4c16-956d-9d1c87951bc1
110282025-04-30T06:43:09.737ZDEBGtest_snapshot_basic (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 56777,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 24,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 97,\n 100,\n 109,\n 105,\n 110,\n 45,\n 107,\n 101,\n 101,\n 112,\n 101,\n 114,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:63319 req_id = 86688c77-5071-4c16-956d-9d1c87951bc1
110292025-04-30T06:43:09.738ZDEBGtest_snapshot_basic (store): query key key = _clickhouse-admin-keeper._tcp
110302025-04-30T06:43:09.738ZERROtest_snapshot_basic (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 56777, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } NXDOMAIN: no records found for name: "_clickhouse-admin-keeper._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:63319 req_id = 86688c77-5071-4c16-956d-9d1c87951bc1
110312025-04-30T06:43:09.738ZTRACtest_snapshot_basic (dropshot_internal): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:61468 method = GET remote_addr = 127.0.0.1:64332 req_id = 08d6672b-37b4-4e25-9771-f72334eddb1c resource = Database uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
110322025-04-30T06:43:09.739ZDEBGtest_snapshot_basic (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:61468 method = GET remote_addr = 127.0.0.1:64332 req_id = 08d6672b-37b4-4e25-9771-f72334eddb1c roles = RoleSet { roles: {} } uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
110332025-04-30T06:43:09.740ZDEBGtest_snapshot_basic (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:61468 method = GET remote_addr = 127.0.0.1:64332 req_id = 08d6672b-37b4-4e25-9771-f72334eddb1c resource = Database result = Ok(()) uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
110342025-04-30T06:43:09.741ZDEBGtest_snapshot_basic (BackgroundTasks): begin collection background_task = inventory_collection
110352025-04-30T06:43:09.741ZDEBGtest_snapshot_basic (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:41502
110362025-04-30T06:43:09.741ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/ignition
110372025-04-30T06:43:09.743ZINFOtest_snapshot_basic (dropshot_internal): request completed latency_us = 6963 local_addr = 127.0.0.1:61468 method = GET remote_addr = 127.0.0.1:64332 req_id = 08d6672b-37b4-4e25-9771-f72334eddb1c response_code = 200 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
110382025-04-30T06:43:09.744ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "d93824c1-9dc6-4c78-9a51-fa2199abf3fb", "content-length": "835", "date": "Wed, 30 Apr 2025 06:43:08 GMT"} })
110392025-04-30T06:43:09.744ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/0
110402025-04-30T06:43:09.744ZDEBGtest_snapshot_basic (oximeter): client response result = Ok(Response { url: "http://127.0.0.1:61468/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending", status: 200, headers: {"content-type": "application/json", "x-request-id": "08d6672b-37b4-4e25-9771-f72334eddb1c", "content-length": "405", "date": "Wed, 30 Apr 2025 06:43:08 GMT"} })
110412025-04-30T06:43:09.745ZDEBGtest_snapshot_basic (oximeter): client request body = None method = GET uri = http://127.0.0.1:61468/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
110422025-04-30T06:43:09.749ZTRACtest_snapshot_basic (dropshot_internal): incoming request local_addr = 127.0.0.1:61468 method = GET remote_addr = 127.0.0.1:64332 req_id = 51fc80fd-2bc4-4f17-9e5e-867db3f93282 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
110432025-04-30T06:43:09.749ZDEBGtest_snapshot_basic (collection-task): sent timer-based collection request to the collection task collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = 354a52d4-3078-4706-81ce-12c32f6bbeaa
110442025-04-30T06:43:09.749ZTRACtest_snapshot_basic (dropshot_internal): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:61468 method = GET remote_addr = 127.0.0.1:64332 req_id = 51fc80fd-2bc4-4f17-9e5e-867db3f93282 resource = Database uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
110452025-04-30T06:43:09.749ZDEBGtest_snapshot_basic (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:61468 method = GET remote_addr = 127.0.0.1:64332 req_id = 51fc80fd-2bc4-4f17-9e5e-867db3f93282 roles = RoleSet { roles: {} } uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
110462025-04-30T06:43:09.750ZDEBGtest_snapshot_basic (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:61468 method = GET remote_addr = 127.0.0.1:64332 req_id = 51fc80fd-2bc4-4f17-9e5e-867db3f93282 resource = Database result = Ok(()) uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
110472025-04-30T06:43:09.750ZDEBGtest_snapshot_basic (collection-task): collecting from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = 354a52d4-3078-4706-81ce-12c32f6bbeaa
110482025-04-30T06:43:09.753ZTRACtest_snapshot_basic (dropshot): incoming request local_addr = 127.0.0.1:47319 method = GET remote_addr = 127.0.0.1:51756 req_id = 01bfb1d6-ac3f-4f50-beab-520f476725d3 uri = /354a52d4-3078-4706-81ce-12c32f6bbeaa
110492025-04-30T06:43:09.762ZINFOtest_snapshot_basic (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 9458 local_addr = 127.0.0.1:47319 method = GET remote_addr = 127.0.0.1:51756 req_id = 01bfb1d6-ac3f-4f50-beab-520f476725d3 response_code = 200 uri = /354a52d4-3078-4706-81ce-12c32f6bbeaa
110502025-04-30T06:43:09.765ZTRACtest_snapshot_basic (collection-task): sent collection request to producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = 354a52d4-3078-4706-81ce-12c32f6bbeaa
110512025-04-30T06:43:09.765ZINFOtest_snapshot_basic (dropshot_internal): request completed latency_us = 16798 local_addr = 127.0.0.1:61468 method = GET remote_addr = 127.0.0.1:64332 req_id = 51fc80fd-2bc4-4f17-9e5e-867db3f93282 response_code = 200 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
110522025-04-30T06:43:09.775ZDEBGtest_snapshot_basic (collection-task): collected results from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 n_results = 5 producer_id = 354a52d4-3078-4706-81ce-12c32f6bbeaa
110532025-04-30T06:43:09.775ZTRACtest_snapshot_basic (collection-task): forwarded results to main collection loop collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = 354a52d4-3078-4706-81ce-12c32f6bbeaa
110542025-04-30T06:43:09.775ZTRACtest_snapshot_basic (collection-task): top of inner collection loop, waiting for next request collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = 354a52d4-3078-4706-81ce-12c32f6bbeaa
110552025-04-30T06:43:09.779ZDEBGtest_snapshot_basic (oximeter): client response result = Ok(Response { url: "http://127.0.0.1:61468/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D", status: 200, headers: {"content-type": "application/json", "x-request-id": "51fc80fd-2bc4-4f17-9e5e-867db3f93282", "content-length": "29", "date": "Wed, 30 Apr 2025 06:43:08 GMT"} })
110562025-04-30T06:43:09.782ZTRACtest_snapshot_basic (oximeter-agent): ignoring request to update existing metric producer, since the endpoint information is the same as the existing collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = 354a52d4-3078-4706-81ce-12c32f6bbeaa
110572025-04-30T06:43:09.782ZTRACtest_snapshot_basic (oximeter-agent): ignoring request to update existing metric producer, since the endpoint information is the same as the existing collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
110582025-04-30T06:43:09.782ZINFOtest_snapshot_basic (oximeter-agent): refreshed list of producers from Nexus collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 file = oximeter/collector/src/agent.rs:637 n_current_tasks = 2 n_pruned_tasks = 0
110592025-04-30T06:43:09.784ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "4358fa7e-8d7e-4c95-987e-69cadb89580a", "content-length": "734", "date": "Wed, 30 Apr 2025 06:43:08 GMT"} })
110602025-04-30T06:43:09.814ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/0/component/sp/caboose?firmware_slot=0
110612025-04-30T06:43:09.817ZDEBGtest_snapshot_basic (collection-task): sent timer-based collection request to the collection task collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
110622025-04-30T06:43:09.818ZDEBGtest_snapshot_basic (collection-task): collecting from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
110632025-04-30T06:43:09.835ZTRACtest_snapshot_basic (collection-task): sent collection request to producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
110642025-04-30T06:43:09.854ZDEBGtest_snapshot_basic (collection-task): collected results from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 n_results = 1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
110652025-04-30T06:43:09.854ZTRACtest_snapshot_basic (collection-task): forwarded results to main collection loop collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
110662025-04-30T06:43:09.854ZTRACtest_snapshot_basic (collection-task): top of inner collection loop, waiting for next request collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
110672025-04-30T06:43:09.854ZDEBGtest_snapshot_basic: attempt to fetch latest inventory collection result = Ok("found")
110682025-04-30T06:43:09.867ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "f478f9f4-5954-496a-bfc6-92a608b3acbb", "content-length": "111", "date": "Wed, 30 Apr 2025 06:43:08 GMT"} })
110692025-04-30T06:43:09.867ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/0/component/sp/caboose?firmware_slot=1
110702025-04-30T06:43:09.869ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "a747fdd7-164d-4639-8923-dd8c6d8a059f", "content-length": "111", "date": "Wed, 30 Apr 2025 06:43:08 GMT"} })
110712025-04-30T06:43:09.869ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/0/component/rot/caboose?firmware_slot=0
110722025-04-30T06:43:09.871ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "378ab705-e37c-4e73-b310-383d753e10a6", "content-length": "170", "date": "Wed, 30 Apr 2025 06:43:08 GMT"} })
110732025-04-30T06:43:09.871ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/0/component/rot/caboose?firmware_slot=1
110742025-04-30T06:43:09.874ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "8a8b1244-8201-4634-8b29-647399dd477d", "content-length": "170", "date": "Wed, 30 Apr 2025 06:43:08 GMT"} })
110752025-04-30T06:43:09.874ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/0/component/stage0/caboose?firmware_slot=0
110762025-04-30T06:43:09.876ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "9184eaec-cccc-4725-9d5f-701c81559277", "content-length": "179", "date": "Wed, 30 Apr 2025 06:43:08 GMT"} })
110772025-04-30T06:43:09.876ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/0/component/stage0/caboose?firmware_slot=1
110782025-04-30T06:43:09.878ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "fcddf07d-3c76-4734-bfe3-380016830944", "content-length": "179", "date": "Wed, 30 Apr 2025 06:43:08 GMT"} })
110792025-04-30T06:43:09.878ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/0/component/rot/cmpa
110802025-04-30T06:43:09.886ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "1afcd8bc-9686-4ff9-bb0c-490659143f59", "content-length": "702", "date": "Wed, 30 Apr 2025 06:43:08 GMT"} })
110812025-04-30T06:43:09.886ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/0/component/rot/cfpa
110822025-04-30T06:43:09.887ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "9bd60347-4c12-4699-b59b-bed71ddcaaf5", "content-length": "727", "date": "Wed, 30 Apr 2025 06:43:08 GMT"} })
110832025-04-30T06:43:09.887ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/0/component/rot/cfpa
110842025-04-30T06:43:09.888ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "7771b4d0-d546-4452-b3fc-21c623b7a2fc", "content-length": "729", "date": "Wed, 30 Apr 2025 06:43:08 GMT"} })
110852025-04-30T06:43:09.888ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/0/component/rot/cfpa
110862025-04-30T06:43:09.890ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "4f96c57c-dacb-4b83-8c22-de72a84924ea", "content-length": "728", "date": "Wed, 30 Apr 2025 06:43:08 GMT"} })
110872025-04-30T06:43:09.891ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/1
110882025-04-30T06:43:09.893ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "0e363368-ca81-4624-84cc-07a24f91a0bf", "content-length": "734", "date": "Wed, 30 Apr 2025 06:43:08 GMT"} })
110892025-04-30T06:43:09.893ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/1/component/sp/caboose?firmware_slot=0
110902025-04-30T06:43:09.900ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "30d6a25f-8ef0-4439-8770-81a9b15c9053", "content-length": "111", "date": "Wed, 30 Apr 2025 06:43:08 GMT"} })
110912025-04-30T06:43:09.901ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/1/component/sp/caboose?firmware_slot=1
110922025-04-30T06:43:09.905ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "3882e92d-8176-4d80-8752-e2aa19114685", "content-length": "111", "date": "Wed, 30 Apr 2025 06:43:08 GMT"} })
110932025-04-30T06:43:09.906ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/1/component/rot/caboose?firmware_slot=0
110942025-04-30T06:43:09.906ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
110952025-04-30T06:43:09.906ZTRACtest_snapshot_basic (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
110962025-04-30T06:43:09.906ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
110972025-04-30T06:43:09.907ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {} }
110982025-04-30T06:43:09.908ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
110992025-04-30T06:43:09.912ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
111002025-04-30T06:43:09.920ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
111012025-04-30T06:43:09.920ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
111022025-04-30T06:43:09.920ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {} }
111032025-04-30T06:43:09.921ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
111042025-04-30T06:43:09.924ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
111052025-04-30T06:43:09.924ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {} }
111062025-04-30T06:43:09.925ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
111072025-04-30T06:43:09.929ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "d314986b-a50f-4553-96fe-1303b1346e73", "content-length": "170", "date": "Wed, 30 Apr 2025 06:43:08 GMT"} })
111082025-04-30T06:43:09.930ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/1/component/rot/caboose?firmware_slot=1
111092025-04-30T06:43:09.971ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "08b8e8f2-de11-4929-a05f-7a8c2e32a17a", "content-length": "170", "date": "Wed, 30 Apr 2025 06:43:09 GMT"} })
111102025-04-30T06:43:09.972ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/1/component/stage0/caboose?firmware_slot=0
111112025-04-30T06:43:09.978ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "31f792a7-50d1-4359-9f78-0787ad2df0b3", "content-length": "179", "date": "Wed, 30 Apr 2025 06:43:09 GMT"} })
111122025-04-30T06:43:09.979ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/1/component/stage0/caboose?firmware_slot=1
111132025-04-30T06:43:09.988ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "70e77041-c51f-4428-b567-75368da13d79", "content-length": "179", "date": "Wed, 30 Apr 2025 06:43:09 GMT"} })
111142025-04-30T06:43:09.988ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/1/component/rot/cmpa
111152025-04-30T06:43:09.989ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "37a081ee-0e50-414b-b62d-c362a13fdf76", "content-length": "702", "date": "Wed, 30 Apr 2025 06:43:09 GMT"} })
111162025-04-30T06:43:09.990ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/1/component/rot/cfpa
111172025-04-30T06:43:09.991ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "08f42a97-d142-4bac-8d4d-4dbd5b44795b", "content-length": "727", "date": "Wed, 30 Apr 2025 06:43:09 GMT"} })
111182025-04-30T06:43:09.991ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/1/component/rot/cfpa
111192025-04-30T06:43:09.998ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "12c6e2ab-0847-471f-84df-5953e0998750", "content-length": "729", "date": "Wed, 30 Apr 2025 06:43:09 GMT"} })
111202025-04-30T06:43:10.004ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/1/component/rot/cfpa
111212025-04-30T06:43:10.004ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "e6565921-dcb5-4aa1-a400-06357201079d", "content-length": "728", "date": "Wed, 30 Apr 2025 06:43:09 GMT"} })
111222025-04-30T06:43:10.005ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/0
111232025-04-30T06:43:10.005ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "937a8ecd-44f2-49ae-850d-7819d69c59cc", "content-length": "723", "date": "Wed, 30 Apr 2025 06:43:09 GMT"} })
111242025-04-30T06:43:10.006ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/0/component/sp/caboose?firmware_slot=0
111252025-04-30T06:43:10.016ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "0859f378-9b58-4b2e-9f1f-f2124a627e43", "content-length": "109", "date": "Wed, 30 Apr 2025 06:43:09 GMT"} })
111262025-04-30T06:43:10.017ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/0/component/sp/caboose?firmware_slot=1
111272025-04-30T06:43:10.042ZDEBGtest_snapshot_basic (results-sink): inserting 6 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
111282025-04-30T06:43:10.043ZDEBGtest_snapshot_basic (results-sink-cluster): inserting 6 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
111292025-04-30T06:43:10.044ZDEBGtest_snapshot_basic (clickhouse-client): unrolling 6 total samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 6ad06c6b-9a6e-4ed1-b3f4-5d2e9da3f8a7
111302025-04-30T06:43:10.053ZTRACtest_snapshot_basic (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 6ad06c6b-9a6e-4ed1-b3f4-5d2e9da3f8a7 n_columns = 4 n_rows = 12 sql = INSERT INTO oximeter.fields_string FORMAT Native
111312025-04-30T06:43:10.055ZDEBGtest_snapshot_basic: attempt to fetch latest inventory collection result = Ok("found")
111322025-04-30T06:43:10.084ZDEBGtest_snapshot_basic (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 6ad06c6b-9a6e-4ed1-b3f4-5d2e9da3f8a7 n_rows = 12 table_name = fields_string
111332025-04-30T06:43:10.084ZTRACtest_snapshot_basic (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 6ad06c6b-9a6e-4ed1-b3f4-5d2e9da3f8a7 n_columns = 4 n_rows = 5 sql = INSERT INTO oximeter.fields_u16 FORMAT Native
111342025-04-30T06:43:10.106ZDEBGtest_snapshot_basic (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 6ad06c6b-9a6e-4ed1-b3f4-5d2e9da3f8a7 n_rows = 5 table_name = fields_u16
111352025-04-30T06:43:10.107ZTRACtest_snapshot_basic (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 6ad06c6b-9a6e-4ed1-b3f4-5d2e9da3f8a7 n_columns = 4 n_rows = 5 sql = INSERT INTO oximeter.fields_uuid FORMAT Native
111362025-04-30T06:43:10.108ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
111372025-04-30T06:43:10.108ZTRACtest_snapshot_basic (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
111382025-04-30T06:43:10.108ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
111392025-04-30T06:43:10.108ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {} }
111402025-04-30T06:43:10.109ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
111412025-04-30T06:43:10.111ZDEBGtest_snapshot_basic (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 6ad06c6b-9a6e-4ed1-b3f4-5d2e9da3f8a7 n_rows = 5 table_name = fields_uuid
111422025-04-30T06:43:10.112ZTRACtest_snapshot_basic (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 6ad06c6b-9a6e-4ed1-b3f4-5d2e9da3f8a7 n_columns = 19 n_rows = 5 sql = INSERT INTO oximeter.measurements_histogramu64 FORMAT Native
111432025-04-30T06:43:10.117ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
111442025-04-30T06:43:10.124ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
111452025-04-30T06:43:10.124ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
111462025-04-30T06:43:10.124ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {} }
111472025-04-30T06:43:10.125ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
111482025-04-30T06:43:10.125ZDEBGtest_snapshot_basic (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 6ad06c6b-9a6e-4ed1-b3f4-5d2e9da3f8a7 n_rows = 5 table_name = measurements_histogramu64
111492025-04-30T06:43:10.126ZTRACtest_snapshot_basic (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 6ad06c6b-9a6e-4ed1-b3f4-5d2e9da3f8a7 n_columns = 4 n_rows = 1 sql = INSERT INTO oximeter.measurements_i64 FORMAT Native
111502025-04-30T06:43:10.128ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "49b5cf4f-bd5d-4b94-9351-485869d47cd7", "content-length": "109", "date": "Wed, 30 Apr 2025 06:43:09 GMT"} })
111512025-04-30T06:43:10.128ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/0/component/rot/caboose?firmware_slot=0
111522025-04-30T06:43:10.128ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
111532025-04-30T06:43:10.128ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {} }
111542025-04-30T06:43:10.130ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
111552025-04-30T06:43:10.130ZDEBGtest_snapshot_basic (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 6ad06c6b-9a6e-4ed1-b3f4-5d2e9da3f8a7 n_rows = 1 table_name = measurements_i64
111562025-04-30T06:43:10.130ZTRACtest_snapshot_basic (results-sink): successfully inserted samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
111572025-04-30T06:43:10.137ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "bbce8b70-1bd6-4b5f-9153-88d183289615", "content-length": "169", "date": "Wed, 30 Apr 2025 06:43:09 GMT"} })
111582025-04-30T06:43:10.137ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/0/component/rot/caboose?firmware_slot=1
111592025-04-30T06:43:10.139ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "fe6ca0fe-64dc-4699-8846-f8cd2dafc8ef", "content-length": "169", "date": "Wed, 30 Apr 2025 06:43:09 GMT"} })
111602025-04-30T06:43:10.140ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/0/component/stage0/caboose?firmware_slot=0
111612025-04-30T06:43:10.142ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "e0ece07f-0e46-44e6-99b6-2a9bb6044bfe", "content-length": "178", "date": "Wed, 30 Apr 2025 06:43:09 GMT"} })
111622025-04-30T06:43:10.142ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/0/component/stage0/caboose?firmware_slot=1
111632025-04-30T06:43:10.144ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "9ceef793-8403-4988-8321-d424223606e5", "content-length": "178", "date": "Wed, 30 Apr 2025 06:43:09 GMT"} })
111642025-04-30T06:43:10.145ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/0/component/rot/cmpa
111652025-04-30T06:43:10.145ZWARNtest_snapshot_basic (results-sink-cluster): failed to insert some results into metric DB collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 error = Failed to check out connection to database: No backends found for this service file = oximeter/collector/src/results_sink.rs:92
111662025-04-30T06:43:10.146ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "a563074f-5df8-495b-b97e-a639e0a2e6bd", "content-length": "702", "date": "Wed, 30 Apr 2025 06:43:09 GMT"} })
111672025-04-30T06:43:10.147ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/0/component/rot/cfpa
111682025-04-30T06:43:10.147ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "d8cbc752-cc13-4b8d-838b-8a28a5a03a83", "content-length": "727", "date": "Wed, 30 Apr 2025 06:43:09 GMT"} })
111692025-04-30T06:43:10.148ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/0/component/rot/cfpa
111702025-04-30T06:43:10.149ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "0803d69a-21ba-4272-9470-618bf97d97da", "content-length": "729", "date": "Wed, 30 Apr 2025 06:43:09 GMT"} })
111712025-04-30T06:43:10.149ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/0/component/rot/cfpa
111722025-04-30T06:43:10.151ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "dabdf1e8-3168-44d8-a0bd-1fae4a38aee3", "content-length": "728", "date": "Wed, 30 Apr 2025 06:43:09 GMT"} })
111732025-04-30T06:43:10.151ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/1
111742025-04-30T06:43:10.153ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "e34b41ca-8ea2-4b8d-b30b-2270301ee7da", "content-length": "723", "date": "Wed, 30 Apr 2025 06:43:09 GMT"} })
111752025-04-30T06:43:10.153ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/1/component/sp/caboose?firmware_slot=0
111762025-04-30T06:43:10.155ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "1e4b4cc9-2788-4832-b05f-04cf641fcc31", "content-length": "109", "date": "Wed, 30 Apr 2025 06:43:09 GMT"} })
111772025-04-30T06:43:10.155ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/1/component/sp/caboose?firmware_slot=1
111782025-04-30T06:43:10.160ZDEBGtest_snapshot_basic: attempt to fetch latest inventory collection result = Ok("found")
111792025-04-30T06:43:10.162ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "e22a15c2-e63f-4852-aed2-271642d4020d", "content-length": "109", "date": "Wed, 30 Apr 2025 06:43:09 GMT"} })
111802025-04-30T06:43:10.162ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/1/component/rot/caboose?firmware_slot=0
111812025-04-30T06:43:10.164ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "75808a58-d02c-4a8f-94ca-280b8da4423c", "content-length": "169", "date": "Wed, 30 Apr 2025 06:43:09 GMT"} })
111822025-04-30T06:43:10.164ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/1/component/rot/caboose?firmware_slot=1
111832025-04-30T06:43:10.166ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "e3dbf813-db8f-44fe-8893-4be750d1ce38", "content-length": "169", "date": "Wed, 30 Apr 2025 06:43:09 GMT"} })
111842025-04-30T06:43:10.166ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/1/component/stage0/caboose?firmware_slot=0
111852025-04-30T06:43:10.168ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "68e5eb9e-0c6e-468d-8f72-05454ffea855", "content-length": "178", "date": "Wed, 30 Apr 2025 06:43:09 GMT"} })
111862025-04-30T06:43:10.168ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/1/component/stage0/caboose?firmware_slot=1
111872025-04-30T06:43:10.170ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "34d34bb7-3ac6-4130-b18d-0d3cf79a6c90", "content-length": "178", "date": "Wed, 30 Apr 2025 06:43:09 GMT"} })
111882025-04-30T06:43:10.171ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/1/component/rot/cmpa
111892025-04-30T06:43:10.172ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "2ac04fc9-3006-486d-b2b3-393167c71a21", "content-length": "702", "date": "Wed, 30 Apr 2025 06:43:09 GMT"} })
111902025-04-30T06:43:10.172ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/1/component/rot/cfpa
111912025-04-30T06:43:10.173ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "335aa36b-f1ad-4393-ab7a-b1535694b012", "content-length": "727", "date": "Wed, 30 Apr 2025 06:43:09 GMT"} })
111922025-04-30T06:43:10.173ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/1/component/rot/cfpa
111932025-04-30T06:43:10.175ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "41fd7e8a-d18a-487c-831d-1861c96e1c51", "content-length": "729", "date": "Wed, 30 Apr 2025 06:43:09 GMT"} })
111942025-04-30T06:43:10.175ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/1/component/rot/cfpa
111952025-04-30T06:43:10.176ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "fc209843-5661-4a49-9adb-a9f3abddc59d", "content-length": "728", "date": "Wed, 30 Apr 2025 06:43:09 GMT"} })
111962025-04-30T06:43:10.176ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
111972025-04-30T06:43:10.176ZTRACtest_snapshot_basic (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
111982025-04-30T06:43:10.176ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
111992025-04-30T06:43:10.176ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
112002025-04-30T06:43:10.177ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
112012025-04-30T06:43:10.182ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
112022025-04-30T06:43:10.186ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
112032025-04-30T06:43:10.187ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
112042025-04-30T06:43:10.187ZTRACtest_snapshot_basic (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
112052025-04-30T06:43:10.187ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
112062025-04-30T06:43:10.187ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
112072025-04-30T06:43:10.187ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
112082025-04-30T06:43:10.207ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
112092025-04-30T06:43:10.212ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
112102025-04-30T06:43:10.212ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
112112025-04-30T06:43:10.212ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
112122025-04-30T06:43:10.215ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
112132025-04-30T06:43:10.218ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
112142025-04-30T06:43:10.218ZTRACtest_snapshot_basic (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
112152025-04-30T06:43:10.218ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
112162025-04-30T06:43:10.218ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {} }
112172025-04-30T06:43:10.219ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
112182025-04-30T06:43:10.290ZDEBGtest_snapshot_basic (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:41923
112192025-04-30T06:43:10.290ZDEBGtest_snapshot_basic (BackgroundTasks): client request SledAgent = http://[::1]:41923 background_task = inventory_collection body = None method = GET uri = http://[::1]:41923/inventory
112202025-04-30T06:43:10.290ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
112212025-04-30T06:43:10.296ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
112222025-04-30T06:43:10.296ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
112232025-04-30T06:43:10.296ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {} }
112242025-04-30T06:43:10.296ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
112252025-04-30T06:43:10.297ZINFOtest_snapshot_basic (dropshot): accepted connection local_addr = [::1]:41923 remote_addr = [::1]:57154 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
112262025-04-30T06:43:10.297ZTRACtest_snapshot_basic (dropshot): incoming request local_addr = [::1]:41923 method = GET remote_addr = [::1]:57154 req_id = 2aea6c6f-7dea-4360-9c3a-43879e856668 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
112272025-04-30T06:43:10.298ZINFOtest_snapshot_basic (dropshot): request completed latency_us = 821 local_addr = [::1]:41923 method = GET remote_addr = [::1]:57154 req_id = 2aea6c6f-7dea-4360-9c3a-43879e856668 response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
112282025-04-30T06:43:10.299ZDEBGtest_snapshot_basic (BackgroundTasks): client response SledAgent = http://[::1]:41923 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:41923/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "2aea6c6f-7dea-4360-9c3a-43879e856668", "content-length": "3141", "date": "Wed, 30 Apr 2025 06:43:09 GMT"} })
112292025-04-30T06:43:10.301ZDEBGtest_snapshot_basic (BackgroundTasks): begin collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
112302025-04-30T06:43:10.301ZDEBGtest_snapshot_basic (BackgroundTasks): end collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
112312025-04-30T06:43:10.301ZDEBGtest_snapshot_basic (BackgroundTasks): finished collection background_task = inventory_collection
112322025-04-30T06:43:10.303ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
112332025-04-30T06:43:10.304ZTRACtest_snapshot_basic (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
112342025-04-30T06:43:10.304ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
112352025-04-30T06:43:10.304ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
112362025-04-30T06:43:10.304ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
112372025-04-30T06:43:10.304ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
112382025-04-30T06:43:10.304ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {} }
112392025-04-30T06:43:10.305ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
112402025-04-30T06:43:10.309ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
112412025-04-30T06:43:10.313ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
112422025-04-30T06:43:10.314ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
112432025-04-30T06:43:10.314ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
112442025-04-30T06:43:10.314ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
112452025-04-30T06:43:10.395ZDEBGtest_snapshot_basic: attempt to fetch latest inventory collection result = Ok("found")
112462025-04-30T06:43:10.413ZWARNtest_snapshot_basic (datastore): Non-retryable transaction failure err = DatabaseError(UniqueViolation, "duplicate key value violates unique constraint \\"inv_caboose_pkey\\"") transaction = inventory_insert_collection
112472025-04-30T06:43:10.414ZWARNtest_snapshot_basic (BackgroundTasks): inventory collection failed background_task = inventory_collection error = failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint "inv_caboose_pkey"
112482025-04-30T06:43:10.414ZDEBGtest_snapshot_basic (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 786.514068ms iteration = 39 status = {"error":"failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint \\"inv_caboose_pkey\\""}
112492025-04-30T06:43:10.414ZDEBGtest_snapshot_basic (BackgroundTasks): activating background_task = inventory_collection iteration = 40 reason = Signaled
112502025-04-30T06:43:10.414ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
112512025-04-30T06:43:10.414ZTRACtest_snapshot_basic (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
112522025-04-30T06:43:10.414ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
112532025-04-30T06:43:10.414ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
112542025-04-30T06:43:10.415ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
112552025-04-30T06:43:10.417ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
112562025-04-30T06:43:10.425ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
112572025-04-30T06:43:10.427ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
112582025-04-30T06:43:10.427ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
112592025-04-30T06:43:10.427ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
112602025-04-30T06:43:10.438ZDEBGtest_snapshot_basic (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = [(d1ad44e5-18dd-4193-a70e-bf9042f0e0e1, 0)]
112612025-04-30T06:43:10.438ZTRACtest_snapshot_basic (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
112622025-04-30T06:43:10.439ZTRACtest_snapshot_basic (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: 0a a1 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 ............._mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:57173 req_id = 5e1158f2-4ce9-43a2-9044-2591bd5c951f
112632025-04-30T06:43:10.439ZDEBGtest_snapshot_basic (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 2721,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:57173 req_id = 5e1158f2-4ce9-43a2-9044-2591bd5c951f
112642025-04-30T06:43:10.439ZDEBGtest_snapshot_basic (store): query key key = _mgs._tcp
112652025-04-30T06:43:10.439ZDEBGtest_snapshot_basic (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
112662025-04-30T06:43:10.439ZDEBGtest_snapshot_basic (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:57173 query = LowerQuery { name: LowerName(Name("_mgs._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 41502, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = 5e1158f2-4ce9-43a2-9044-2591bd5c951f
112672025-04-30T06:43:10.440ZDEBGtest_snapshot_basic (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 41502, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 4255, tv_nsec: 239325826 } })
112682025-04-30T06:43:10.440ZTRACtest_snapshot_basic (DnsResolver): lookup_service_targets: looking up SRV target
112692025-04-30T06:43:10.441ZTRACtest_snapshot_basic (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: 70 a9 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e p...........-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:50798 req_id = 152bebaa-9b71-4aa8-b5f9-dd9621f4fbe2
112702025-04-30T06:43:10.441ZDEBGtest_snapshot_basic (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 28841,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:50798 req_id = 152bebaa-9b71-4aa8-b5f9-dd9621f4fbe2
112712025-04-30T06:43:10.441ZDEBGtest_snapshot_basic (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
112722025-04-30T06:43:10.441ZDEBGtest_snapshot_basic (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:50798 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = 152bebaa-9b71-4aa8-b5f9-dd9621f4fbe2
112732025-04-30T06:43:10.523ZTRACtest_snapshot_basic (DnsResolver): lookup_all_socket_v6 srv dns_name = _clickhouse-admin-keeper._tcp.control-plane.oxide.internal
112742025-04-30T06:43:10.524ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
112752025-04-30T06:43:10.524ZTRACtest_snapshot_basic (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
112762025-04-30T06:43:10.524ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
112772025-04-30T06:43:10.524ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {} }
112782025-04-30T06:43:10.526ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
112792025-04-30T06:43:10.532ZTRACtest_snapshot_basic (dns): buffer buffer = Length: 87 (0x57) bytes\n0000: ba d6 01 00 00 01 00 00 00 00 00 01 18 5f 63 6c ............._cl\n0010: 69 63 6b 68 6f 75 73 65 2d 61 64 6d 69 6e 2d 6b ickhouse-admin-k\n0020: 65 65 70 65 72 04 5f 74 63 70 0d 63 6f 6e 74 72 eeper._tcp.contr\n0030: 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 69 ol-plane.oxide.i\n0040: 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 04 nternal..!....).\n0050: d0 00 00 00 00 00 00 ....... kind = dns peer_addr = [::1]:61649 req_id = 08e7bd03-5730-40b8-96df-4fdb38b4c74f
112802025-04-30T06:43:10.532ZDEBGtest_snapshot_basic (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 47830,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 24,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 97,\n 100,\n 109,\n 105,\n 110,\n 45,\n 107,\n 101,\n 101,\n 112,\n 101,\n 114,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:61649 req_id = 08e7bd03-5730-40b8-96df-4fdb38b4c74f
112812025-04-30T06:43:10.532ZDEBGtest_snapshot_basic (store): query key key = _clickhouse-admin-keeper._tcp
112822025-04-30T06:43:10.532ZERROtest_snapshot_basic (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 47830, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } NXDOMAIN: no records found for name: "_clickhouse-admin-keeper._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:61649 req_id = 08e7bd03-5730-40b8-96df-4fdb38b4c74f
112832025-04-30T06:43:10.548ZDEBGtest_snapshot_basic (BackgroundTasks): begin collection background_task = inventory_collection
112842025-04-30T06:43:10.548ZDEBGtest_snapshot_basic (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:41502
112852025-04-30T06:43:10.548ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/ignition
112862025-04-30T06:43:10.549ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
112872025-04-30T06:43:10.555ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
112882025-04-30T06:43:10.555ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
112892025-04-30T06:43:10.555ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {} }
112902025-04-30T06:43:10.556ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
112912025-04-30T06:43:10.572ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
112922025-04-30T06:43:10.572ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {} }
112932025-04-30T06:43:10.573ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
112942025-04-30T06:43:10.593ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "6621b1ad-3c51-4c46-8b15-f3835ca49c57", "content-length": "835", "date": "Wed, 30 Apr 2025 06:43:09 GMT"} })
112952025-04-30T06:43:10.593ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/0
112962025-04-30T06:43:10.650ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "a440aabe-ff10-4eb8-a099-062fd23260db", "content-length": "734", "date": "Wed, 30 Apr 2025 06:43:09 GMT"} })
112972025-04-30T06:43:10.650ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/0/component/sp/caboose?firmware_slot=0
112982025-04-30T06:43:10.667ZTRACtest_snapshot_basic (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal
112992025-04-30T06:43:10.692ZTRACtest_snapshot_basic (dns): buffer buffer = Length: 72 (0x48) bytes\n0000: 80 92 01 00 00 01 00 00 00 00 00 01 09 5f 64 65 ............._de\n0010: 6e 64 72 69 74 65 04 5f 74 63 70 0d 63 6f 6e 74 ndrite._tcp.cont\n0020: 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 rol-plane.oxide.\n0030: 69 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 internal..!....)\n0040: 04 d0 00 00 00 00 00 00 ........ kind = dns peer_addr = [::1]:55486 req_id = 4953c96d-6660-4bdc-b7dd-48c882e905f4
113002025-04-30T06:43:10.692ZDEBGtest_snapshot_basic (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 32914,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_dendrite._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_dendrite._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 9,\n 95,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:55486 req_id = 4953c96d-6660-4bdc-b7dd-48c882e905f4
113012025-04-30T06:43:10.693ZDEBGtest_snapshot_basic (store): query key key = _dendrite._tcp
113022025-04-30T06:43:10.693ZDEBGtest_snapshot_basic (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
113032025-04-30T06:43:10.693ZDEBGtest_snapshot_basic (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:55486 query = LowerQuery { name: LowerName(Name("_dendrite._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 32976, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = 4953c96d-6660-4bdc-b7dd-48c882e905f4
113042025-04-30T06:43:10.694ZDEBGtest_snapshot_basic (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 32976, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 4255, tv_nsec: 493552957 } })
113052025-04-30T06:43:10.694ZTRACtest_snapshot_basic (DnsResolver): lookup_service_targets: looking up SRV target
113062025-04-30T06:43:10.695ZTRACtest_snapshot_basic (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: 41 ea 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e A...........-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:53651 req_id = fbcc077e-1e4c-4cf0-ad3c-f86e5be21323
113072025-04-30T06:43:10.695ZDEBGtest_snapshot_basic (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 16874,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:53651 req_id = fbcc077e-1e4c-4cf0-ad3c-f86e5be21323
113082025-04-30T06:43:10.695ZDEBGtest_snapshot_basic (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
113092025-04-30T06:43:10.695ZDEBGtest_snapshot_basic (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:53651 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = fbcc077e-1e4c-4cf0-ad3c-f86e5be21323
113102025-04-30T06:43:10.696ZINFOtest_snapshot_basic (BackgroundTasks): Determining switch slots managed by switch zones background_task = nat_v4_garbage_collector
113112025-04-30T06:43:10.786ZINFOtest_snapshot_basic (BackgroundTasks): determining switch slot managed by dendrite zone background_task = nat_v4_garbage_collector zone_address = ::1
113122025-04-30T06:43:10.786ZDEBGtest_snapshot_basic (MgsClient): client request background_task = nat_v4_garbage_collector body = None method = GET uri = http://[::1]:12225/local/switch-id
113132025-04-30T06:43:10.786ZDEBGtest_snapshot_basic (MgsClient): client response background_task = nat_v4_garbage_collector result = Err(reqwest::Error { kind: Request, url: "http://[::1]:12225/local/switch-id", source: hyper_util::client::legacy::Error(Connect, ConnectError("tcp connect error", Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) })
113142025-04-30T06:43:10.786ZWARNtest_snapshot_basic (BackgroundTasks): failed to identify switch slot for dendrite background_task = nat_v4_garbage_collector reason = Communication Error: error sending request for url (http://[::1]:12225/local/switch-id): client error (Connect): tcp connect error: Connection refused (os error 146): Connection refused (os error 146) zone_address = ::1
113152025-04-30T06:43:10.788ZWARNtest_snapshot_basic (BackgroundTasks): Failed to map switch zone addr: Communication Error: error sending request for url (http://[::1]:12225/local/switch-id), retrying background_task = nat_v4_garbage_collector
113162025-04-30T06:43:10.788ZTRACtest_snapshot_basic (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal
113172025-04-30T06:43:10.790ZTRACtest_snapshot_basic (dns): buffer buffer = Length: 72 (0x48) bytes\n0000: d8 ce 01 00 00 01 00 00 00 00 00 01 09 5f 64 65 ............._de\n0010: 6e 64 72 69 74 65 04 5f 74 63 70 0d 63 6f 6e 74 ndrite._tcp.cont\n0020: 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 rol-plane.oxide.\n0030: 69 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 internal..!....)\n0040: 04 d0 00 00 00 00 00 00 ........ kind = dns peer_addr = [::1]:58515 req_id = 10145433-fc5c-4c68-9af4-fcf46d699bdc
113182025-04-30T06:43:10.790ZDEBGtest_snapshot_basic (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 55502,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_dendrite._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_dendrite._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 9,\n 95,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:58515 req_id = 10145433-fc5c-4c68-9af4-fcf46d699bdc
113192025-04-30T06:43:10.791ZDEBGtest_snapshot_basic (store): query key key = _dendrite._tcp
113202025-04-30T06:43:10.791ZDEBGtest_snapshot_basic (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
113212025-04-30T06:43:10.791ZDEBGtest_snapshot_basic (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:58515 query = LowerQuery { name: LowerName(Name("_dendrite._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 32976, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = 10145433-fc5c-4c68-9af4-fcf46d699bdc
113222025-04-30T06:43:10.794ZDEBGtest_snapshot_basic (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 32976, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 4255, tv_nsec: 592910159 } })
113232025-04-30T06:43:10.794ZTRACtest_snapshot_basic (DnsResolver): lookup_service_targets: looking up SRV target
113242025-04-30T06:43:10.798ZDEBGtest_snapshot_basic (collection-task): sent timer-based collection request to the collection task collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
113252025-04-30T06:43:10.798ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "15599cbb-cf12-4697-95b9-684d3c6a60c9", "content-length": "111", "date": "Wed, 30 Apr 2025 06:43:09 GMT"} })
113262025-04-30T06:43:10.798ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/0/component/sp/caboose?firmware_slot=1
113272025-04-30T06:43:10.799ZDEBGtest_snapshot_basic (collection-task): collecting from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
113282025-04-30T06:43:10.799ZTRACtest_snapshot_basic (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: db 8b 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e ............-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:57636 req_id = b1585aab-74fc-480c-a092-e70745cade52
113292025-04-30T06:43:10.799ZDEBGtest_snapshot_basic (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 56203,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:57636 req_id = b1585aab-74fc-480c-a092-e70745cade52
113302025-04-30T06:43:10.800ZDEBGtest_snapshot_basic (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
113312025-04-30T06:43:10.800ZDEBGtest_snapshot_basic (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:57636 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = b1585aab-74fc-480c-a092-e70745cade52
113322025-04-30T06:43:10.803ZDEBGtest_snapshot_basic: attempt to fetch latest inventory collection result = Ok("found")
113332025-04-30T06:43:10.804ZINFOtest_snapshot_basic (BackgroundTasks): Determining switch slots managed by switch zones background_task = bfd_manager
113342025-04-30T06:43:10.881ZINFOtest_snapshot_basic (BackgroundTasks): determining switch slot managed by dendrite zone background_task = bfd_manager zone_address = ::1
113352025-04-30T06:43:10.882ZDEBGtest_snapshot_basic (MgsClient): client request background_task = bfd_manager body = None method = GET uri = http://[::1]:12225/local/switch-id
113362025-04-30T06:43:10.882ZDEBGtest_snapshot_basic (MgsClient): client response background_task = bfd_manager result = Err(reqwest::Error { kind: Request, url: "http://[::1]:12225/local/switch-id", source: hyper_util::client::legacy::Error(Connect, ConnectError("tcp connect error", Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) })
113372025-04-30T06:43:10.882ZWARNtest_snapshot_basic (BackgroundTasks): failed to identify switch slot for dendrite background_task = bfd_manager reason = Communication Error: error sending request for url (http://[::1]:12225/local/switch-id): client error (Connect): tcp connect error: Connection refused (os error 146): Connection refused (os error 146) zone_address = ::1
113382025-04-30T06:43:10.883ZWARNtest_snapshot_basic (BackgroundTasks): Failed to map switch zone addr: Communication Error: error sending request for url (http://[::1]:12225/local/switch-id), retrying background_task = bfd_manager
113392025-04-30T06:43:10.883ZTRACtest_snapshot_basic (collection-task): sent collection request to producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
113402025-04-30T06:43:10.885ZDEBGtest_snapshot_basic (collection-task): collected results from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 n_results = 1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
113412025-04-30T06:43:10.885ZTRACtest_snapshot_basic (collection-task): forwarded results to main collection loop collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
113422025-04-30T06:43:10.885ZTRACtest_snapshot_basic (collection-task): top of inner collection loop, waiting for next request collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
113432025-04-30T06:43:10.885ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
113442025-04-30T06:43:10.886ZTRACtest_snapshot_basic (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
113452025-04-30T06:43:10.886ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
113462025-04-30T06:43:10.886ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {} }
113472025-04-30T06:43:10.886ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
113482025-04-30T06:43:10.887ZTRACtest_snapshot_basic (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal
113492025-04-30T06:43:10.888ZTRACtest_snapshot_basic (dns): buffer buffer = Length: 72 (0x48) bytes\n0000: 75 6d 01 00 00 01 00 00 00 00 00 01 09 5f 64 65 um..........._de\n0010: 6e 64 72 69 74 65 04 5f 74 63 70 0d 63 6f 6e 74 ndrite._tcp.cont\n0020: 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 rol-plane.oxide.\n0030: 69 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 internal..!....)\n0040: 04 d0 00 00 00 00 00 00 ........ kind = dns peer_addr = [::1]:56481 req_id = d943b2e5-d4e4-4e9d-bbe1-de28e4b62049
113502025-04-30T06:43:10.888ZDEBGtest_snapshot_basic (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 30061,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_dendrite._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_dendrite._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 9,\n 95,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:56481 req_id = d943b2e5-d4e4-4e9d-bbe1-de28e4b62049
113512025-04-30T06:43:10.888ZDEBGtest_snapshot_basic (store): query key key = _dendrite._tcp
113522025-04-30T06:43:10.888ZDEBGtest_snapshot_basic (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
113532025-04-30T06:43:10.888ZDEBGtest_snapshot_basic (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:56481 query = LowerQuery { name: LowerName(Name("_dendrite._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 32976, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = d943b2e5-d4e4-4e9d-bbe1-de28e4b62049
113542025-04-30T06:43:10.889ZDEBGtest_snapshot_basic (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 32976, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 4255, tv_nsec: 688286881 } })
113552025-04-30T06:43:10.889ZTRACtest_snapshot_basic (DnsResolver): lookup_service_targets: looking up SRV target
113562025-04-30T06:43:10.890ZTRACtest_snapshot_basic (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: b2 13 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e ............-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:52719 req_id = 7b97a059-fc9c-4c77-a930-58a8fef70561
113572025-04-30T06:43:10.890ZDEBGtest_snapshot_basic (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 45587,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:52719 req_id = 7b97a059-fc9c-4c77-a930-58a8fef70561
113582025-04-30T06:43:10.890ZDEBGtest_snapshot_basic (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
113592025-04-30T06:43:10.890ZDEBGtest_snapshot_basic (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:52719 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = 7b97a059-fc9c-4c77-a930-58a8fef70561
113602025-04-30T06:43:10.890ZINFOtest_snapshot_basic (BackgroundTasks): Determining switch slots managed by switch zones background_task = switch_port_config_manager rack_id = c19a698f-c6f9-4a17-ae30-20d711b8f7dc
113612025-04-30T06:43:10.946ZINFOtest_snapshot_basic (BackgroundTasks): determining switch slot managed by dendrite zone background_task = switch_port_config_manager rack_id = c19a698f-c6f9-4a17-ae30-20d711b8f7dc zone_address = ::1
113622025-04-30T06:43:10.946ZDEBGtest_snapshot_basic (MgsClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = c19a698f-c6f9-4a17-ae30-20d711b8f7dc uri = http://[::1]:12225/local/switch-id
113632025-04-30T06:43:10.946ZDEBGtest_snapshot_basic (MgsClient): client response background_task = switch_port_config_manager rack_id = c19a698f-c6f9-4a17-ae30-20d711b8f7dc result = Err(reqwest::Error { kind: Request, url: "http://[::1]:12225/local/switch-id", source: hyper_util::client::legacy::Error(Connect, ConnectError("tcp connect error", Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) })
113642025-04-30T06:43:10.946ZWARNtest_snapshot_basic (BackgroundTasks): failed to identify switch slot for dendrite background_task = switch_port_config_manager rack_id = c19a698f-c6f9-4a17-ae30-20d711b8f7dc reason = Communication Error: error sending request for url (http://[::1]:12225/local/switch-id): client error (Connect): tcp connect error: Connection refused (os error 146): Connection refused (os error 146) zone_address = ::1
113652025-04-30T06:43:10.948ZWARNtest_snapshot_basic (BackgroundTasks): Failed to map switch zone addr: Communication Error: error sending request for url (http://[::1]:12225/local/switch-id), retrying background_task = switch_port_config_manager rack_id = c19a698f-c6f9-4a17-ae30-20d711b8f7dc
113662025-04-30T06:43:10.949ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
113672025-04-30T06:43:10.955ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
113682025-04-30T06:43:10.955ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
113692025-04-30T06:43:10.955ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {} }
113702025-04-30T06:43:10.955ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
113712025-04-30T06:43:10.957ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "6e957382-a9c0-4a7c-afad-4fe31603f2be", "content-length": "111", "date": "Wed, 30 Apr 2025 06:43:09 GMT"} })
113722025-04-30T06:43:10.961ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/0/component/rot/caboose?firmware_slot=0
113732025-04-30T06:43:10.968ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
113742025-04-30T06:43:10.968ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {} }
113752025-04-30T06:43:10.968ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
113762025-04-30T06:43:10.972ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "6538b2ea-c156-435e-81df-6eb3a41d2973", "content-length": "170", "date": "Wed, 30 Apr 2025 06:43:10 GMT"} })
113772025-04-30T06:43:10.972ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/0/component/rot/caboose?firmware_slot=1
113782025-04-30T06:43:10.979ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "e1157a37-e8e3-403d-957c-039f000cebc4", "content-length": "170", "date": "Wed, 30 Apr 2025 06:43:10 GMT"} })
113792025-04-30T06:43:10.979ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/0/component/stage0/caboose?firmware_slot=0
113802025-04-30T06:43:11.035ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "6eded711-1a02-41d2-a5d8-e0ddea7633c6", "content-length": "179", "date": "Wed, 30 Apr 2025 06:43:10 GMT"} })
113812025-04-30T06:43:11.035ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/0/component/stage0/caboose?firmware_slot=1
113822025-04-30T06:43:11.036ZDEBGtest_snapshot_basic (results-sink): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
113832025-04-30T06:43:11.037ZDEBGtest_snapshot_basic (results-sink-cluster): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
113842025-04-30T06:43:11.037ZDEBGtest_snapshot_basic (clickhouse-client): unrolling 1 total samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 6ad06c6b-9a6e-4ed1-b3f4-5d2e9da3f8a7
113852025-04-30T06:43:11.042ZTRACtest_snapshot_basic (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 6ad06c6b-9a6e-4ed1-b3f4-5d2e9da3f8a7 n_columns = 4 n_rows = 2 sql = INSERT INTO oximeter.fields_string FORMAT Native
113862025-04-30T06:43:11.051ZDEBGtest_snapshot_basic (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 6ad06c6b-9a6e-4ed1-b3f4-5d2e9da3f8a7 n_rows = 2 table_name = fields_string
113872025-04-30T06:43:11.052ZTRACtest_snapshot_basic (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 6ad06c6b-9a6e-4ed1-b3f4-5d2e9da3f8a7 n_columns = 4 n_rows = 1 sql = INSERT INTO oximeter.measurements_i64 FORMAT Native
113882025-04-30T06:43:11.059ZDEBGtest_snapshot_basic (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 6ad06c6b-9a6e-4ed1-b3f4-5d2e9da3f8a7 n_rows = 1 table_name = measurements_i64
113892025-04-30T06:43:11.059ZTRACtest_snapshot_basic (results-sink): successfully inserted samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
113902025-04-30T06:43:11.062ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "774e7cd7-7ba3-486f-96c7-fbde96f39ef3", "content-length": "179", "date": "Wed, 30 Apr 2025 06:43:10 GMT"} })
113912025-04-30T06:43:11.062ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/0/component/rot/cmpa
113922025-04-30T06:43:11.063ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "4477bd92-8475-424c-a667-71db1d22e7bc", "content-length": "702", "date": "Wed, 30 Apr 2025 06:43:10 GMT"} })
113932025-04-30T06:43:11.063ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/0/component/rot/cfpa
113942025-04-30T06:43:11.067ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "b94d9bdc-a26d-4bc8-b8a8-61b51dd8d3e9", "content-length": "727", "date": "Wed, 30 Apr 2025 06:43:10 GMT"} })
113952025-04-30T06:43:11.067ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/0/component/rot/cfpa
113962025-04-30T06:43:11.068ZDEBGtest_snapshot_basic: attempt to fetch latest inventory collection result = Ok("found")
113972025-04-30T06:43:11.071ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "62f9ca0d-7f99-4eb1-84de-a6d04e698a50", "content-length": "729", "date": "Wed, 30 Apr 2025 06:43:10 GMT"} })
113982025-04-30T06:43:11.071ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/0/component/rot/cfpa
113992025-04-30T06:43:11.072ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "5d11ecc2-3557-479f-b6ee-883c806c0ba0", "content-length": "728", "date": "Wed, 30 Apr 2025 06:43:10 GMT"} })
114002025-04-30T06:43:11.072ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/1
114012025-04-30T06:43:11.073ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "afd21135-13f9-4c9b-9e27-04b365f2255a", "content-length": "734", "date": "Wed, 30 Apr 2025 06:43:10 GMT"} })
114022025-04-30T06:43:11.074ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/1/component/sp/caboose?firmware_slot=0
114032025-04-30T06:43:11.076ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "2dd4ec09-3280-41e5-a0cb-09c208c953cd", "content-length": "111", "date": "Wed, 30 Apr 2025 06:43:10 GMT"} })
114042025-04-30T06:43:11.076ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/1/component/sp/caboose?firmware_slot=1
114052025-04-30T06:43:11.079ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "aefae8d2-6600-4ddc-9e7f-2120d5bd2ddb", "content-length": "111", "date": "Wed, 30 Apr 2025 06:43:10 GMT"} })
114062025-04-30T06:43:11.079ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/1/component/rot/caboose?firmware_slot=0
114072025-04-30T06:43:11.081ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "97513a6b-1e79-448d-9c72-5873a9ccfdb8", "content-length": "170", "date": "Wed, 30 Apr 2025 06:43:10 GMT"} })
114082025-04-30T06:43:11.084ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/1/component/rot/caboose?firmware_slot=1
114092025-04-30T06:43:11.091ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "452f3395-3d24-4a60-aee2-6e1da72d6d13", "content-length": "170", "date": "Wed, 30 Apr 2025 06:43:10 GMT"} })
114102025-04-30T06:43:11.092ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/1/component/stage0/caboose?firmware_slot=0
114112025-04-30T06:43:11.098ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "aad8f13e-4dd7-44de-b45e-a3fb6ed3fc0f", "content-length": "179", "date": "Wed, 30 Apr 2025 06:43:10 GMT"} })
114122025-04-30T06:43:11.098ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/1/component/stage0/caboose?firmware_slot=1
114132025-04-30T06:43:11.103ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "7458e1f7-ad89-4192-8f3b-3a14843645b5", "content-length": "179", "date": "Wed, 30 Apr 2025 06:43:10 GMT"} })
114142025-04-30T06:43:11.103ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/1/component/rot/cmpa
114152025-04-30T06:43:11.104ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "66ca9c79-e9d1-4799-9bee-2665db2ae915", "content-length": "702", "date": "Wed, 30 Apr 2025 06:43:10 GMT"} })
114162025-04-30T06:43:11.104ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/1/component/rot/cfpa
114172025-04-30T06:43:11.106ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "11de5389-6ca4-48e4-a940-d99b6d375797", "content-length": "727", "date": "Wed, 30 Apr 2025 06:43:10 GMT"} })
114182025-04-30T06:43:11.106ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/1/component/rot/cfpa
114192025-04-30T06:43:11.107ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "c7943ca8-2012-4599-99cb-8eb9ae57784e", "content-length": "729", "date": "Wed, 30 Apr 2025 06:43:10 GMT"} })
114202025-04-30T06:43:11.107ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/1/component/rot/cfpa
114212025-04-30T06:43:11.109ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "db5bd8f9-2adb-4ab6-99fe-edb6e1d7480c", "content-length": "728", "date": "Wed, 30 Apr 2025 06:43:10 GMT"} })
114222025-04-30T06:43:11.109ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/0
114232025-04-30T06:43:11.111ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "79e3e64a-5aef-41e0-9cf7-64cb9a29f9f3", "content-length": "723", "date": "Wed, 30 Apr 2025 06:43:10 GMT"} })
114242025-04-30T06:43:11.111ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/0/component/sp/caboose?firmware_slot=0
114252025-04-30T06:43:11.116ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "89025e96-78f5-43c9-bd09-cc48dce5c927", "content-length": "109", "date": "Wed, 30 Apr 2025 06:43:10 GMT"} })
114262025-04-30T06:43:11.117ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/0/component/sp/caboose?firmware_slot=1
114272025-04-30T06:43:11.118ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
114282025-04-30T06:43:11.118ZTRACtest_snapshot_basic (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
114292025-04-30T06:43:11.118ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
114302025-04-30T06:43:11.119ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {} }
114312025-04-30T06:43:11.120ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
114322025-04-30T06:43:11.121ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "7b10086e-3a5b-4306-aa1f-4992548fded8", "content-length": "109", "date": "Wed, 30 Apr 2025 06:43:10 GMT"} })
114332025-04-30T06:43:11.121ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/0/component/rot/caboose?firmware_slot=0
114342025-04-30T06:43:11.123ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
114352025-04-30T06:43:11.166ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
114362025-04-30T06:43:11.166ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
114372025-04-30T06:43:11.166ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {} }
114382025-04-30T06:43:11.175ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
114392025-04-30T06:43:11.176ZWARNtest_snapshot_basic (results-sink-cluster): failed to insert some results into metric DB collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 error = Failed to check out connection to database: No backends found for this service file = oximeter/collector/src/results_sink.rs:92
114402025-04-30T06:43:11.185ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
114412025-04-30T06:43:11.185ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {} }
114422025-04-30T06:43:11.186ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
114432025-04-30T06:43:11.186ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "62b22544-991c-4fb1-b164-7a3b85dab1a2", "content-length": "169", "date": "Wed, 30 Apr 2025 06:43:10 GMT"} })
114442025-04-30T06:43:11.187ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/0/component/rot/caboose?firmware_slot=1
114452025-04-30T06:43:11.194ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "64d9ff60-c634-4ffd-84b9-519bafa16253", "content-length": "169", "date": "Wed, 30 Apr 2025 06:43:10 GMT"} })
114462025-04-30T06:43:11.194ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/0/component/stage0/caboose?firmware_slot=0
114472025-04-30T06:43:11.199ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "b941643c-1a1c-47c3-843d-536a2fe63a51", "content-length": "178", "date": "Wed, 30 Apr 2025 06:43:10 GMT"} })
114482025-04-30T06:43:11.199ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/0/component/stage0/caboose?firmware_slot=1
114492025-04-30T06:43:11.209ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "af7aabce-14e3-4ad6-a67d-b361eecf8a7b", "content-length": "178", "date": "Wed, 30 Apr 2025 06:43:10 GMT"} })
114502025-04-30T06:43:11.210ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/0/component/rot/cmpa
114512025-04-30T06:43:11.211ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "4b088c66-4da7-48b6-9f25-be56e06f8812", "content-length": "702", "date": "Wed, 30 Apr 2025 06:43:10 GMT"} })
114522025-04-30T06:43:11.212ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/0/component/rot/cfpa
114532025-04-30T06:43:11.213ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "b298ba24-948b-4308-9eac-833c89455ef1", "content-length": "727", "date": "Wed, 30 Apr 2025 06:43:10 GMT"} })
114542025-04-30T06:43:11.213ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/0/component/rot/cfpa
114552025-04-30T06:43:11.215ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "1ba6a426-ff39-4e39-b1ec-404d2364686e", "content-length": "729", "date": "Wed, 30 Apr 2025 06:43:10 GMT"} })
114562025-04-30T06:43:11.215ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/0/component/rot/cfpa
114572025-04-30T06:43:11.218ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "d6a2a2b0-5e55-40eb-9767-98bf9a3e2a87", "content-length": "728", "date": "Wed, 30 Apr 2025 06:43:10 GMT"} })
114582025-04-30T06:43:11.218ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/1
114592025-04-30T06:43:11.221ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "42655ee3-62d8-492c-9756-640ce44abeea", "content-length": "723", "date": "Wed, 30 Apr 2025 06:43:10 GMT"} })
114602025-04-30T06:43:11.221ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/1/component/sp/caboose?firmware_slot=0
114612025-04-30T06:43:11.224ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "69bf8ef6-9f89-44a7-b89c-81fe4c6a6fa4", "content-length": "109", "date": "Wed, 30 Apr 2025 06:43:10 GMT"} })
114622025-04-30T06:43:11.224ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/1/component/sp/caboose?firmware_slot=1
114632025-04-30T06:43:11.228ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "cf166e11-5db6-4c2b-b097-9b082996ad13", "content-length": "109", "date": "Wed, 30 Apr 2025 06:43:10 GMT"} })
114642025-04-30T06:43:11.228ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/1/component/rot/caboose?firmware_slot=0
114652025-04-30T06:43:11.229ZDEBGtest_snapshot_basic: attempt to fetch latest inventory collection result = Ok("found")
114662025-04-30T06:43:11.235ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "1d6e6296-6e46-4d41-8059-98727321a9bd", "content-length": "169", "date": "Wed, 30 Apr 2025 06:43:10 GMT"} })
114672025-04-30T06:43:11.235ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/1/component/rot/caboose?firmware_slot=1
114682025-04-30T06:43:11.241ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "82693b05-2165-48e8-938b-8ebee8f8c0ed", "content-length": "169", "date": "Wed, 30 Apr 2025 06:43:10 GMT"} })
114692025-04-30T06:43:11.256ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/1/component/stage0/caboose?firmware_slot=0
114702025-04-30T06:43:11.264ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "4d6b2b05-45f0-4685-90ff-81238a2059a5", "content-length": "178", "date": "Wed, 30 Apr 2025 06:43:10 GMT"} })
114712025-04-30T06:43:11.264ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/1/component/stage0/caboose?firmware_slot=1
114722025-04-30T06:43:11.266ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "9ed527c0-0fd8-423a-82a3-93745371da58", "content-length": "178", "date": "Wed, 30 Apr 2025 06:43:10 GMT"} })
114732025-04-30T06:43:11.266ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/1/component/rot/cmpa
114742025-04-30T06:43:11.266ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "ef288ff7-4f61-4a1f-a03a-29f5c8096749", "content-length": "702", "date": "Wed, 30 Apr 2025 06:43:10 GMT"} })
114752025-04-30T06:43:11.266ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/1/component/rot/cfpa
114762025-04-30T06:43:11.267ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "ba1e693d-8f19-41e5-9083-0762f3d93e1b", "content-length": "727", "date": "Wed, 30 Apr 2025 06:43:10 GMT"} })
114772025-04-30T06:43:11.268ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/1/component/rot/cfpa
114782025-04-30T06:43:11.273ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "9c9536a8-387a-40e4-bbfe-92210252e628", "content-length": "729", "date": "Wed, 30 Apr 2025 06:43:10 GMT"} })
114792025-04-30T06:43:11.273ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/1/component/rot/cfpa
114802025-04-30T06:43:11.275ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "6d3f9cf1-21be-4465-858a-e3dfcdfca511", "content-length": "728", "date": "Wed, 30 Apr 2025 06:43:10 GMT"} })
114812025-04-30T06:43:11.275ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
114822025-04-30T06:43:11.275ZTRACtest_snapshot_basic (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
114832025-04-30T06:43:11.275ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
114842025-04-30T06:43:11.275ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
114852025-04-30T06:43:11.277ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
114862025-04-30T06:43:11.280ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
114872025-04-30T06:43:11.280ZTRACtest_snapshot_basic (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
114882025-04-30T06:43:11.280ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
114892025-04-30T06:43:11.280ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {} }
114902025-04-30T06:43:11.281ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
114912025-04-30T06:43:11.296ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
114922025-04-30T06:43:11.302ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
114932025-04-30T06:43:11.302ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
114942025-04-30T06:43:11.302ZTRACtest_snapshot_basic (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
114952025-04-30T06:43:11.302ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
114962025-04-30T06:43:11.302ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
114972025-04-30T06:43:11.302ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
114982025-04-30T06:43:11.304ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
114992025-04-30T06:43:11.310ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
115002025-04-30T06:43:11.311ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
115012025-04-30T06:43:11.311ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {} }
115022025-04-30T06:43:11.311ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
115032025-04-30T06:43:11.313ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
115042025-04-30T06:43:11.318ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
115052025-04-30T06:43:11.318ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
115062025-04-30T06:43:11.318ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
115072025-04-30T06:43:11.319ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
115082025-04-30T06:43:11.326ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
115092025-04-30T06:43:11.326ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {} }
115102025-04-30T06:43:11.327ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
115112025-04-30T06:43:11.405ZDEBGtest_snapshot_basic (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:41923
115122025-04-30T06:43:11.405ZDEBGtest_snapshot_basic (BackgroundTasks): client request SledAgent = http://[::1]:41923 background_task = inventory_collection body = None method = GET uri = http://[::1]:41923/inventory
115132025-04-30T06:43:11.408ZINFOtest_snapshot_basic (dropshot): accepted connection local_addr = [::1]:41923 remote_addr = [::1]:39282 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
115142025-04-30T06:43:11.409ZTRACtest_snapshot_basic (dropshot): incoming request local_addr = [::1]:41923 method = GET remote_addr = [::1]:39282 req_id = dae938bb-5d1e-4dd6-ac0c-767156c4d8db sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
115152025-04-30T06:43:11.409ZINFOtest_snapshot_basic (dropshot): request completed latency_us = 519 local_addr = [::1]:41923 method = GET remote_addr = [::1]:39282 req_id = dae938bb-5d1e-4dd6-ac0c-767156c4d8db response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
115162025-04-30T06:43:11.410ZDEBGtest_snapshot_basic (BackgroundTasks): client response SledAgent = http://[::1]:41923 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:41923/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "dae938bb-5d1e-4dd6-ac0c-767156c4d8db", "content-length": "3141", "date": "Wed, 30 Apr 2025 06:43:10 GMT"} })
115172025-04-30T06:43:11.413ZDEBGtest_snapshot_basic (BackgroundTasks): begin collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
115182025-04-30T06:43:11.413ZDEBGtest_snapshot_basic (BackgroundTasks): end collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
115192025-04-30T06:43:11.413ZDEBGtest_snapshot_basic (BackgroundTasks): finished collection background_task = inventory_collection
115202025-04-30T06:43:11.418ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
115212025-04-30T06:43:11.418ZTRACtest_snapshot_basic (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
115222025-04-30T06:43:11.418ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
115232025-04-30T06:43:11.418ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
115242025-04-30T06:43:11.420ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
115252025-04-30T06:43:11.424ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
115262025-04-30T06:43:11.434ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
115272025-04-30T06:43:11.434ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
115282025-04-30T06:43:11.435ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
115292025-04-30T06:43:11.435ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
115302025-04-30T06:43:11.478ZDEBGtest_snapshot_basic: attempt to fetch latest inventory collection result = Ok("found")
115312025-04-30T06:43:11.478ZWARNtest_snapshot_basic (datastore): Non-retryable transaction failure err = DatabaseError(UniqueViolation, "duplicate key value violates unique constraint \\"inv_caboose_pkey\\"") transaction = inventory_insert_collection
115322025-04-30T06:43:11.479ZWARNtest_snapshot_basic (BackgroundTasks): inventory collection failed background_task = inventory_collection error = failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint "inv_caboose_pkey"
115332025-04-30T06:43:11.479ZDEBGtest_snapshot_basic (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 1.064798241s iteration = 40 status = {"error":"failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint \\"inv_caboose_pkey\\""}
115342025-04-30T06:43:11.479ZDEBGtest_snapshot_basic (BackgroundTasks): activating background_task = inventory_collection iteration = 41 reason = Signaled
115352025-04-30T06:43:11.479ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
115362025-04-30T06:43:11.479ZTRACtest_snapshot_basic (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
115372025-04-30T06:43:11.479ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
115382025-04-30T06:43:11.479ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
115392025-04-30T06:43:11.480ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
115402025-04-30T06:43:11.495ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
115412025-04-30T06:43:11.500ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
115422025-04-30T06:43:11.500ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
115432025-04-30T06:43:11.500ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
115442025-04-30T06:43:11.500ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
115452025-04-30T06:43:11.505ZDEBGtest_snapshot_basic (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = [(d1ad44e5-18dd-4193-a70e-bf9042f0e0e1, 0)]
115462025-04-30T06:43:11.505ZTRACtest_snapshot_basic (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
115472025-04-30T06:43:11.506ZTRACtest_snapshot_basic (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: f9 a7 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 ............._mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:61392 req_id = 0bca8a92-c28e-452d-8ad8-08af0fbaa24c
115482025-04-30T06:43:11.506ZDEBGtest_snapshot_basic (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 63911,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:61392 req_id = 0bca8a92-c28e-452d-8ad8-08af0fbaa24c
115492025-04-30T06:43:11.506ZDEBGtest_snapshot_basic (store): query key key = _mgs._tcp
115502025-04-30T06:43:11.507ZDEBGtest_snapshot_basic (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
115512025-04-30T06:43:11.507ZDEBGtest_snapshot_basic (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:61392 query = LowerQuery { name: LowerName(Name("_mgs._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 41502, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = 0bca8a92-c28e-452d-8ad8-08af0fbaa24c
115522025-04-30T06:43:11.507ZDEBGtest_snapshot_basic (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 41502, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 4256, tv_nsec: 306602905 } })
115532025-04-30T06:43:11.507ZTRACtest_snapshot_basic (DnsResolver): lookup_service_targets: looking up SRV target
115542025-04-30T06:43:11.509ZTRACtest_snapshot_basic (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: 56 0b 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e V...........-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:54322 req_id = cb5d1e22-498d-4ca4-84db-c1217c441e32
115552025-04-30T06:43:11.509ZDEBGtest_snapshot_basic (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 22027,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:54322 req_id = cb5d1e22-498d-4ca4-84db-c1217c441e32
115562025-04-30T06:43:11.509ZDEBGtest_snapshot_basic (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
115572025-04-30T06:43:11.509ZDEBGtest_snapshot_basic (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:54322 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = cb5d1e22-498d-4ca4-84db-c1217c441e32
115582025-04-30T06:43:11.568ZTRACtest_snapshot_basic (DnsResolver): lookup_all_socket_v6 srv dns_name = _clickhouse-admin-keeper._tcp.control-plane.oxide.internal
115592025-04-30T06:43:11.569ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
115602025-04-30T06:43:11.569ZTRACtest_snapshot_basic (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
115612025-04-30T06:43:11.569ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
115622025-04-30T06:43:11.569ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {} }
115632025-04-30T06:43:11.570ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
115642025-04-30T06:43:11.575ZTRACtest_snapshot_basic (dns): buffer buffer = Length: 87 (0x57) bytes\n0000: 83 18 01 00 00 01 00 00 00 00 00 01 18 5f 63 6c ............._cl\n0010: 69 63 6b 68 6f 75 73 65 2d 61 64 6d 69 6e 2d 6b ickhouse-admin-k\n0020: 65 65 70 65 72 04 5f 74 63 70 0d 63 6f 6e 74 72 eeper._tcp.contr\n0030: 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 69 ol-plane.oxide.i\n0040: 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 04 nternal..!....).\n0050: d0 00 00 00 00 00 00 ....... kind = dns peer_addr = [::1]:63810 req_id = 596cb40c-3f14-4338-8a21-d17bb752c94c
115652025-04-30T06:43:11.575ZDEBGtest_snapshot_basic (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 33560,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 24,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 97,\n 100,\n 109,\n 105,\n 110,\n 45,\n 107,\n 101,\n 101,\n 112,\n 101,\n 114,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:63810 req_id = 596cb40c-3f14-4338-8a21-d17bb752c94c
115662025-04-30T06:43:11.576ZDEBGtest_snapshot_basic (store): query key key = _clickhouse-admin-keeper._tcp
115672025-04-30T06:43:11.576ZERROtest_snapshot_basic (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 33560, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } NXDOMAIN: no records found for name: "_clickhouse-admin-keeper._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:63810 req_id = 596cb40c-3f14-4338-8a21-d17bb752c94c
115682025-04-30T06:43:11.576ZDEBGtest_snapshot_basic (BackgroundTasks): begin collection background_task = inventory_collection
115692025-04-30T06:43:11.576ZDEBGtest_snapshot_basic (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:41502
115702025-04-30T06:43:11.576ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/ignition
115712025-04-30T06:43:11.582ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
115722025-04-30T06:43:11.590ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
115732025-04-30T06:43:11.590ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
115742025-04-30T06:43:11.590ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {} }
115752025-04-30T06:43:11.590ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
115762025-04-30T06:43:11.601ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
115772025-04-30T06:43:11.601ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {} }
115782025-04-30T06:43:11.601ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
115792025-04-30T06:43:11.602ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "a6b1a3c4-80c1-48ea-bf6e-9f8df0b78468", "content-length": "835", "date": "Wed, 30 Apr 2025 06:43:10 GMT"} })
115802025-04-30T06:43:11.603ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/0
115812025-04-30T06:43:11.626ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "9337db9a-8458-4636-b731-fa07176958ea", "content-length": "734", "date": "Wed, 30 Apr 2025 06:43:10 GMT"} })
115822025-04-30T06:43:11.627ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/0/component/sp/caboose?firmware_slot=0
115832025-04-30T06:43:11.635ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "79bc5eba-abdd-403c-b93c-8381e21769c5", "content-length": "111", "date": "Wed, 30 Apr 2025 06:43:10 GMT"} })
115842025-04-30T06:43:11.635ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/0/component/sp/caboose?firmware_slot=1
115852025-04-30T06:43:11.637ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "62bf586b-d811-40c5-a624-81c360fc0f21", "content-length": "111", "date": "Wed, 30 Apr 2025 06:43:10 GMT"} })
115862025-04-30T06:43:11.638ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/0/component/rot/caboose?firmware_slot=0
115872025-04-30T06:43:11.642ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "ff9cae84-cd2f-4b36-8029-a8342afffa09", "content-length": "170", "date": "Wed, 30 Apr 2025 06:43:10 GMT"} })
115882025-04-30T06:43:11.642ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/0/component/rot/caboose?firmware_slot=1
115892025-04-30T06:43:11.646ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "c5511ab4-d69b-452f-b0f2-fc45f7dc15f6", "content-length": "170", "date": "Wed, 30 Apr 2025 06:43:10 GMT"} })
115902025-04-30T06:43:11.647ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/0/component/stage0/caboose?firmware_slot=0
115912025-04-30T06:43:11.650ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "c6a1c116-7182-47f4-9a54-61d14a4dd660", "content-length": "179", "date": "Wed, 30 Apr 2025 06:43:10 GMT"} })
115922025-04-30T06:43:11.651ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/0/component/stage0/caboose?firmware_slot=1
115932025-04-30T06:43:11.654ZDEBGtest_snapshot_basic: attempt to fetch latest inventory collection result = Ok("found")
115942025-04-30T06:43:11.656ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "4d816449-242a-4d9a-b38a-bd991ba5b730", "content-length": "179", "date": "Wed, 30 Apr 2025 06:43:10 GMT"} })
115952025-04-30T06:43:11.656ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/0/component/rot/cmpa
115962025-04-30T06:43:11.658ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "b4d6a6b5-ce3a-47bf-83be-ea46a8f05885", "content-length": "702", "date": "Wed, 30 Apr 2025 06:43:10 GMT"} })
115972025-04-30T06:43:11.658ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/0/component/rot/cfpa
115982025-04-30T06:43:11.660ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "866f11e0-5151-45ce-9a10-3f2524879fad", "content-length": "727", "date": "Wed, 30 Apr 2025 06:43:10 GMT"} })
115992025-04-30T06:43:11.660ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/0/component/rot/cfpa
116002025-04-30T06:43:11.662ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "06f3c777-beb0-4000-a811-bc5cba76d55f", "content-length": "729", "date": "Wed, 30 Apr 2025 06:43:10 GMT"} })
116012025-04-30T06:43:11.662ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/0/component/rot/cfpa
116022025-04-30T06:43:11.664ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "1f8638f5-9ced-4f4d-b9ea-2ac704c8ef79", "content-length": "728", "date": "Wed, 30 Apr 2025 06:43:10 GMT"} })
116032025-04-30T06:43:11.664ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/1
116042025-04-30T06:43:11.666ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "b9898fcf-91a4-409c-b59d-754f2eb908d9", "content-length": "734", "date": "Wed, 30 Apr 2025 06:43:10 GMT"} })
116052025-04-30T06:43:11.666ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/1/component/sp/caboose?firmware_slot=0
116062025-04-30T06:43:11.676ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "a2a9be50-12d6-40ca-aafc-fd731c9a9287", "content-length": "111", "date": "Wed, 30 Apr 2025 06:43:10 GMT"} })
116072025-04-30T06:43:11.677ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/1/component/sp/caboose?firmware_slot=1
116082025-04-30T06:43:11.680ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "10e7bd2a-0a8b-4b2a-864b-ebc08bc15c77", "content-length": "111", "date": "Wed, 30 Apr 2025 06:43:10 GMT"} })
116092025-04-30T06:43:11.680ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/1/component/rot/caboose?firmware_slot=0
116102025-04-30T06:43:11.685ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "974e36bc-3db0-404f-bd9c-75f2aec0eb1f", "content-length": "170", "date": "Wed, 30 Apr 2025 06:43:10 GMT"} })
116112025-04-30T06:43:11.686ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/1/component/rot/caboose?firmware_slot=1
116122025-04-30T06:43:11.688ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "1dedf269-2a9d-461b-b7b9-4903ba1dec39", "content-length": "170", "date": "Wed, 30 Apr 2025 06:43:10 GMT"} })
116132025-04-30T06:43:11.688ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/1/component/stage0/caboose?firmware_slot=0
116142025-04-30T06:43:11.693ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "2efe9e20-58c5-4b2a-9281-b6c011608a96", "content-length": "179", "date": "Wed, 30 Apr 2025 06:43:10 GMT"} })
116152025-04-30T06:43:11.694ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/1/component/stage0/caboose?firmware_slot=1
116162025-04-30T06:43:11.695ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "fc5617af-24b8-4c3d-ad08-15100debe7b3", "content-length": "179", "date": "Wed, 30 Apr 2025 06:43:10 GMT"} })
116172025-04-30T06:43:11.696ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/1/component/rot/cmpa
116182025-04-30T06:43:11.697ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "96e0243e-12b0-441c-b2af-57fe73b141e7", "content-length": "702", "date": "Wed, 30 Apr 2025 06:43:10 GMT"} })
116192025-04-30T06:43:11.697ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/1/component/rot/cfpa
116202025-04-30T06:43:11.698ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "64c82dd4-330d-45a3-bd32-cbb046d021ad", "content-length": "727", "date": "Wed, 30 Apr 2025 06:43:10 GMT"} })
116212025-04-30T06:43:11.698ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/1/component/rot/cfpa
116222025-04-30T06:43:11.699ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "31c8fe5b-87cf-42e7-a9d4-53f8b67f85b2", "content-length": "729", "date": "Wed, 30 Apr 2025 06:43:10 GMT"} })
116232025-04-30T06:43:11.700ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/1/component/rot/cfpa
116242025-04-30T06:43:11.701ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "e0a285e9-01b1-47e7-a433-c662e0a6cdfa", "content-length": "728", "date": "Wed, 30 Apr 2025 06:43:10 GMT"} })
116252025-04-30T06:43:11.701ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/0
116262025-04-30T06:43:11.704ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "96ddd6f7-7b40-4736-ae32-8d2896fe1b42", "content-length": "723", "date": "Wed, 30 Apr 2025 06:43:10 GMT"} })
116272025-04-30T06:43:11.704ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/0/component/sp/caboose?firmware_slot=0
116282025-04-30T06:43:11.704ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
116292025-04-30T06:43:11.705ZTRACtest_snapshot_basic (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
116302025-04-30T06:43:11.705ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
116312025-04-30T06:43:11.705ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {} }
116322025-04-30T06:43:11.706ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
116332025-04-30T06:43:11.708ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
116342025-04-30T06:43:11.716ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
116352025-04-30T06:43:11.716ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
116362025-04-30T06:43:11.716ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {} }
116372025-04-30T06:43:11.717ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
116382025-04-30T06:43:11.719ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "92d2fc82-8fb9-4ce5-b727-2223f07e305a", "content-length": "109", "date": "Wed, 30 Apr 2025 06:43:10 GMT"} })
116392025-04-30T06:43:11.719ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/0/component/sp/caboose?firmware_slot=1
116402025-04-30T06:43:11.721ZINFOtest_snapshot_basic (oximeter-agent): refreshing list of producers from Nexus collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 file = oximeter/collector/src/agent.rs:562
116412025-04-30T06:43:11.721ZDEBGtest_snapshot_basic (oximeter): client request body = None method = GET uri = http://127.0.0.1:61468/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
116422025-04-30T06:43:11.721ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
116432025-04-30T06:43:11.721ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {} }
116442025-04-30T06:43:11.722ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
116452025-04-30T06:43:11.729ZTRACtest_snapshot_basic (dropshot_internal): incoming request local_addr = 127.0.0.1:61468 method = GET remote_addr = 127.0.0.1:64332 req_id = 5bf9e302-9067-481e-897f-46df828ab77f uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
116462025-04-30T06:43:11.732ZTRACtest_snapshot_basic (dropshot_internal): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:61468 method = GET remote_addr = 127.0.0.1:64332 req_id = 5bf9e302-9067-481e-897f-46df828ab77f resource = Database uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
116472025-04-30T06:43:11.734ZDEBGtest_snapshot_basic (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:61468 method = GET remote_addr = 127.0.0.1:64332 req_id = 5bf9e302-9067-481e-897f-46df828ab77f roles = RoleSet { roles: {} } uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
116482025-04-30T06:43:11.735ZDEBGtest_snapshot_basic (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:61468 method = GET remote_addr = 127.0.0.1:64332 req_id = 5bf9e302-9067-481e-897f-46df828ab77f resource = Database result = Ok(()) uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
116492025-04-30T06:43:11.739ZINFOtest_snapshot_basic (dropshot_internal): request completed latency_us = 17189 local_addr = 127.0.0.1:61468 method = GET remote_addr = 127.0.0.1:64332 req_id = 5bf9e302-9067-481e-897f-46df828ab77f response_code = 200 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
116502025-04-30T06:43:11.740ZDEBGtest_snapshot_basic (oximeter): client response result = Ok(Response { url: "http://127.0.0.1:61468/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending", status: 200, headers: {"content-type": "application/json", "x-request-id": "5bf9e302-9067-481e-897f-46df828ab77f", "content-length": "405", "date": "Wed, 30 Apr 2025 06:43:10 GMT"} })
116512025-04-30T06:43:11.741ZDEBGtest_snapshot_basic (oximeter): client request body = None method = GET uri = http://127.0.0.1:61468/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
116522025-04-30T06:43:11.741ZTRACtest_snapshot_basic (dropshot_internal): incoming request local_addr = 127.0.0.1:61468 method = GET remote_addr = 127.0.0.1:64332 req_id = 701e8bd1-fb32-4433-8c67-8574796d5aef uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
116532025-04-30T06:43:11.742ZTRACtest_snapshot_basic (dropshot_internal): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:61468 method = GET remote_addr = 127.0.0.1:64332 req_id = 701e8bd1-fb32-4433-8c67-8574796d5aef resource = Database uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
116542025-04-30T06:43:11.742ZDEBGtest_snapshot_basic (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:61468 method = GET remote_addr = 127.0.0.1:64332 req_id = 701e8bd1-fb32-4433-8c67-8574796d5aef roles = RoleSet { roles: {} } uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
116552025-04-30T06:43:11.742ZDEBGtest_snapshot_basic (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:61468 method = GET remote_addr = 127.0.0.1:64332 req_id = 701e8bd1-fb32-4433-8c67-8574796d5aef resource = Database result = Ok(()) uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
116562025-04-30T06:43:11.744ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "23f85671-db65-42f2-858d-8b556eeffa3c", "content-length": "109", "date": "Wed, 30 Apr 2025 06:43:10 GMT"} })
116572025-04-30T06:43:11.744ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/0/component/rot/caboose?firmware_slot=0
116582025-04-30T06:43:11.747ZINFOtest_snapshot_basic (dropshot_internal): request completed latency_us = 5486 local_addr = 127.0.0.1:61468 method = GET remote_addr = 127.0.0.1:64332 req_id = 701e8bd1-fb32-4433-8c67-8574796d5aef response_code = 200 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
116592025-04-30T06:43:11.748ZDEBGtest_snapshot_basic (oximeter): client response result = Ok(Response { url: "http://127.0.0.1:61468/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D", status: 200, headers: {"content-type": "application/json", "x-request-id": "701e8bd1-fb32-4433-8c67-8574796d5aef", "content-length": "29", "date": "Wed, 30 Apr 2025 06:43:10 GMT"} })
116602025-04-30T06:43:11.748ZTRACtest_snapshot_basic (oximeter-agent): ignoring request to update existing metric producer, since the endpoint information is the same as the existing collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = 354a52d4-3078-4706-81ce-12c32f6bbeaa
116612025-04-30T06:43:11.749ZTRACtest_snapshot_basic (oximeter-agent): ignoring request to update existing metric producer, since the endpoint information is the same as the existing collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
116622025-04-30T06:43:11.749ZINFOtest_snapshot_basic (oximeter-agent): refreshed list of producers from Nexus collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 file = oximeter/collector/src/agent.rs:637 n_current_tasks = 2 n_pruned_tasks = 0
116632025-04-30T06:43:11.753ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "faab1a4d-2104-47b4-9af4-b365f09d093a", "content-length": "169", "date": "Wed, 30 Apr 2025 06:43:10 GMT"} })
116642025-04-30T06:43:11.753ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/0/component/rot/caboose?firmware_slot=1
116652025-04-30T06:43:11.757ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "d2217b19-bfaa-426f-80cd-b831958cd6f9", "content-length": "169", "date": "Wed, 30 Apr 2025 06:43:10 GMT"} })
116662025-04-30T06:43:11.758ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/0/component/stage0/caboose?firmware_slot=0
116672025-04-30T06:43:11.761ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "a2ed9d1e-091f-4ebe-9f7f-fc4c3ebc9976", "content-length": "178", "date": "Wed, 30 Apr 2025 06:43:10 GMT"} })
116682025-04-30T06:43:11.761ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/0/component/stage0/caboose?firmware_slot=1
116692025-04-30T06:43:11.765ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "b8da26d3-b620-4871-933c-06377120b00f", "content-length": "178", "date": "Wed, 30 Apr 2025 06:43:10 GMT"} })
116702025-04-30T06:43:11.767ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/0/component/rot/cmpa
116712025-04-30T06:43:11.769ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "aa571913-82a1-4fec-a61e-be5a742403e5", "content-length": "702", "date": "Wed, 30 Apr 2025 06:43:10 GMT"} })
116722025-04-30T06:43:11.769ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/0/component/rot/cfpa
116732025-04-30T06:43:11.771ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "a0c561cb-1e2e-4fee-8b78-eb06d1621315", "content-length": "727", "date": "Wed, 30 Apr 2025 06:43:10 GMT"} })
116742025-04-30T06:43:11.771ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/0/component/rot/cfpa
116752025-04-30T06:43:11.772ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "1503f0c2-efc5-4ecb-8a32-58ecc2e73f34", "content-length": "729", "date": "Wed, 30 Apr 2025 06:43:10 GMT"} })
116762025-04-30T06:43:11.772ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/0/component/rot/cfpa
116772025-04-30T06:43:11.777ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "6b4b9a3a-4b18-4848-a8e7-4befec707302", "content-length": "728", "date": "Wed, 30 Apr 2025 06:43:10 GMT"} })
116782025-04-30T06:43:11.777ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/1
116792025-04-30T06:43:11.777ZDEBGtest_snapshot_basic: attempt to fetch latest inventory collection result = Ok("found")
116802025-04-30T06:43:11.779ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "52ba85c3-2fc3-4b3f-b56c-34f8e5f941b9", "content-length": "723", "date": "Wed, 30 Apr 2025 06:43:10 GMT"} })
116812025-04-30T06:43:11.779ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/1/component/sp/caboose?firmware_slot=0
116822025-04-30T06:43:11.782ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "62a7518b-5d59-45ee-b046-6d73b6f98af0", "content-length": "109", "date": "Wed, 30 Apr 2025 06:43:10 GMT"} })
116832025-04-30T06:43:11.782ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/1/component/sp/caboose?firmware_slot=1
116842025-04-30T06:43:11.785ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "3f36cbca-8d08-4e8a-8229-a64532fe3cd1", "content-length": "109", "date": "Wed, 30 Apr 2025 06:43:10 GMT"} })
116852025-04-30T06:43:11.785ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/1/component/rot/caboose?firmware_slot=0
116862025-04-30T06:43:11.787ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "9cccb09b-c87b-4902-8502-742c18f6ce53", "content-length": "169", "date": "Wed, 30 Apr 2025 06:43:10 GMT"} })
116872025-04-30T06:43:11.787ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/1/component/rot/caboose?firmware_slot=1
116882025-04-30T06:43:11.790ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "a088b0a0-c307-4c77-b22e-c7edb3affc33", "content-length": "169", "date": "Wed, 30 Apr 2025 06:43:10 GMT"} })
116892025-04-30T06:43:11.790ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/1/component/stage0/caboose?firmware_slot=0
116902025-04-30T06:43:11.798ZDEBGtest_snapshot_basic (collection-task): sent timer-based collection request to the collection task collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
116912025-04-30T06:43:11.798ZDEBGtest_snapshot_basic (collection-task): collecting from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
116922025-04-30T06:43:11.803ZTRACtest_snapshot_basic (collection-task): sent collection request to producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
116932025-04-30T06:43:11.803ZDEBGtest_snapshot_basic (collection-task): collected results from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 n_results = 1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
116942025-04-30T06:43:11.803ZTRACtest_snapshot_basic (collection-task): forwarded results to main collection loop collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
116952025-04-30T06:43:11.803ZTRACtest_snapshot_basic (collection-task): top of inner collection loop, waiting for next request collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
116962025-04-30T06:43:11.805ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "49b7848d-8e72-44c1-84b3-84a3b4bd7c9d", "content-length": "178", "date": "Wed, 30 Apr 2025 06:43:10 GMT"} })
116972025-04-30T06:43:11.806ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/1/component/stage0/caboose?firmware_slot=1
116982025-04-30T06:43:11.808ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "ef896248-3b77-47ec-9346-f8180c4b7882", "content-length": "178", "date": "Wed, 30 Apr 2025 06:43:10 GMT"} })
116992025-04-30T06:43:11.808ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/1/component/rot/cmpa
117002025-04-30T06:43:11.811ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "2a083097-5203-4403-9bc2-444c0dac16fc", "content-length": "702", "date": "Wed, 30 Apr 2025 06:43:10 GMT"} })
117012025-04-30T06:43:11.811ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/1/component/rot/cfpa
117022025-04-30T06:43:11.811ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "f5412ea6-c264-4b6c-9371-456ecb2d31cb", "content-length": "727", "date": "Wed, 30 Apr 2025 06:43:10 GMT"} })
117032025-04-30T06:43:11.812ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/1/component/rot/cfpa
117042025-04-30T06:43:11.813ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "87d046bd-9e18-4006-aabd-34928f317000", "content-length": "729", "date": "Wed, 30 Apr 2025 06:43:10 GMT"} })
117052025-04-30T06:43:11.814ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/1/component/rot/cfpa
117062025-04-30T06:43:11.815ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "8838349e-1cb6-4d04-857a-ec132a124937", "content-length": "728", "date": "Wed, 30 Apr 2025 06:43:10 GMT"} })
117072025-04-30T06:43:11.815ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
117082025-04-30T06:43:11.815ZTRACtest_snapshot_basic (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
117092025-04-30T06:43:11.815ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
117102025-04-30T06:43:11.815ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
117112025-04-30T06:43:11.816ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
117122025-04-30T06:43:11.818ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
117132025-04-30T06:43:11.822ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
117142025-04-30T06:43:11.822ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
117152025-04-30T06:43:11.822ZTRACtest_snapshot_basic (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
117162025-04-30T06:43:11.822ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
117172025-04-30T06:43:11.822ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
117182025-04-30T06:43:11.823ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
117192025-04-30T06:43:11.828ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
117202025-04-30T06:43:11.832ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
117212025-04-30T06:43:11.832ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
117222025-04-30T06:43:11.832ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
117232025-04-30T06:43:11.833ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
117242025-04-30T06:43:11.833ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
117252025-04-30T06:43:11.833ZTRACtest_snapshot_basic (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
117262025-04-30T06:43:11.833ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
117272025-04-30T06:43:11.833ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {} }
117282025-04-30T06:43:11.834ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
117292025-04-30T06:43:11.893ZDEBGtest_snapshot_basic (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:41923
117302025-04-30T06:43:11.893ZDEBGtest_snapshot_basic (BackgroundTasks): client request SledAgent = http://[::1]:41923 background_task = inventory_collection body = None method = GET uri = http://[::1]:41923/inventory
117312025-04-30T06:43:11.894ZINFOtest_snapshot_basic (dropshot): accepted connection local_addr = [::1]:41923 remote_addr = [::1]:48743 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
117322025-04-30T06:43:11.894ZTRACtest_snapshot_basic (dropshot): incoming request local_addr = [::1]:41923 method = GET remote_addr = [::1]:48743 req_id = 988c39a9-2a8d-4365-bc35-dc4c7d13e98d sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
117332025-04-30T06:43:11.896ZINFOtest_snapshot_basic (dropshot): request completed latency_us = 1447 local_addr = [::1]:41923 method = GET remote_addr = [::1]:48743 req_id = 988c39a9-2a8d-4365-bc35-dc4c7d13e98d response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
117342025-04-30T06:43:11.896ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
117352025-04-30T06:43:11.903ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
117362025-04-30T06:43:11.903ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
117372025-04-30T06:43:11.903ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {} }
117382025-04-30T06:43:11.904ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
117392025-04-30T06:43:11.904ZDEBGtest_snapshot_basic (BackgroundTasks): client response SledAgent = http://[::1]:41923 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:41923/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "988c39a9-2a8d-4365-bc35-dc4c7d13e98d", "content-length": "3141", "date": "Wed, 30 Apr 2025 06:43:10 GMT"} })
117402025-04-30T06:43:11.911ZDEBGtest_snapshot_basic (BackgroundTasks): begin collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
117412025-04-30T06:43:11.911ZDEBGtest_snapshot_basic (BackgroundTasks): end collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
117422025-04-30T06:43:11.911ZDEBGtest_snapshot_basic (BackgroundTasks): finished collection background_task = inventory_collection
117432025-04-30T06:43:11.915ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
117442025-04-30T06:43:11.915ZTRACtest_snapshot_basic (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
117452025-04-30T06:43:11.915ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
117462025-04-30T06:43:11.915ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
117472025-04-30T06:43:11.916ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
117482025-04-30T06:43:11.920ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
117492025-04-30T06:43:11.920ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {} }
117502025-04-30T06:43:11.921ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
117512025-04-30T06:43:11.923ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
117522025-04-30T06:43:11.932ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
117532025-04-30T06:43:11.932ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
117542025-04-30T06:43:11.932ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
117552025-04-30T06:43:11.933ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
117562025-04-30T06:43:11.990ZDEBGtest_snapshot_basic: attempt to fetch latest inventory collection result = Ok("found")
117572025-04-30T06:43:12.009ZWARNtest_snapshot_basic (datastore): Non-retryable transaction failure err = DatabaseError(UniqueViolation, "duplicate key value violates unique constraint \\"inv_caboose_pkey\\"") transaction = inventory_insert_collection
117582025-04-30T06:43:12.009ZWARNtest_snapshot_basic (BackgroundTasks): inventory collection failed background_task = inventory_collection error = failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint "inv_caboose_pkey"
117592025-04-30T06:43:12.009ZDEBGtest_snapshot_basic (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 530.46734ms iteration = 41 status = {"error":"failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint \\"inv_caboose_pkey\\""}
117602025-04-30T06:43:12.009ZDEBGtest_snapshot_basic (BackgroundTasks): activating background_task = inventory_collection iteration = 42 reason = Signaled
117612025-04-30T06:43:12.009ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
117622025-04-30T06:43:12.009ZTRACtest_snapshot_basic (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
117632025-04-30T06:43:12.009ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
117642025-04-30T06:43:12.010ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
117652025-04-30T06:43:12.011ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
117662025-04-30T06:43:12.037ZDEBGtest_snapshot_basic (results-sink): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
117672025-04-30T06:43:12.038ZDEBGtest_snapshot_basic (results-sink-cluster): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
117682025-04-30T06:43:12.038ZDEBGtest_snapshot_basic (clickhouse-client): unrolling 1 total samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 6ad06c6b-9a6e-4ed1-b3f4-5d2e9da3f8a7
117692025-04-30T06:43:12.039ZTRACtest_snapshot_basic (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 6ad06c6b-9a6e-4ed1-b3f4-5d2e9da3f8a7 n_columns = 4 n_rows = 2 sql = INSERT INTO oximeter.fields_string FORMAT Native
117702025-04-30T06:43:12.039ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
117712025-04-30T06:43:12.044ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
117722025-04-30T06:43:12.044ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
117732025-04-30T06:43:12.044ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
117742025-04-30T06:43:12.044ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
117752025-04-30T06:43:12.046ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
117762025-04-30T06:43:12.046ZTRACtest_snapshot_basic (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
117772025-04-30T06:43:12.046ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
117782025-04-30T06:43:12.046ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {} }
117792025-04-30T06:43:12.047ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
117802025-04-30T06:43:12.051ZDEBGtest_snapshot_basic (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 6ad06c6b-9a6e-4ed1-b3f4-5d2e9da3f8a7 n_rows = 2 table_name = fields_string
117812025-04-30T06:43:12.051ZTRACtest_snapshot_basic (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 6ad06c6b-9a6e-4ed1-b3f4-5d2e9da3f8a7 n_columns = 4 n_rows = 1 sql = INSERT INTO oximeter.measurements_i64 FORMAT Native
117822025-04-30T06:43:12.051ZDEBGtest_snapshot_basic (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = [(d1ad44e5-18dd-4193-a70e-bf9042f0e0e1, 0)]
117832025-04-30T06:43:12.051ZTRACtest_snapshot_basic (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
117842025-04-30T06:43:12.053ZTRACtest_snapshot_basic (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: 54 17 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 T............_mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:51119 req_id = c1fdde3b-ccd1-47c6-afd5-1f74ee453b71
117852025-04-30T06:43:12.053ZDEBGtest_snapshot_basic (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 21527,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:51119 req_id = c1fdde3b-ccd1-47c6-afd5-1f74ee453b71
117862025-04-30T06:43:12.053ZDEBGtest_snapshot_basic (store): query key key = _mgs._tcp
117872025-04-30T06:43:12.053ZDEBGtest_snapshot_basic (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
117882025-04-30T06:43:12.053ZDEBGtest_snapshot_basic (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:51119 query = LowerQuery { name: LowerName(Name("_mgs._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 41502, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = c1fdde3b-ccd1-47c6-afd5-1f74ee453b71
117892025-04-30T06:43:12.055ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
117902025-04-30T06:43:12.061ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
117912025-04-30T06:43:12.061ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
117922025-04-30T06:43:12.061ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {} }
117932025-04-30T06:43:12.073ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
117942025-04-30T06:43:12.073ZDEBGtest_snapshot_basic (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 41502, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 4256, tv_nsec: 872764855 } })
117952025-04-30T06:43:12.074ZTRACtest_snapshot_basic (DnsResolver): lookup_service_targets: looking up SRV target
117962025-04-30T06:43:12.074ZDEBGtest_snapshot_basic (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 6ad06c6b-9a6e-4ed1-b3f4-5d2e9da3f8a7 n_rows = 1 table_name = measurements_i64
117972025-04-30T06:43:12.074ZTRACtest_snapshot_basic (results-sink): successfully inserted samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
117982025-04-30T06:43:12.082ZTRACtest_snapshot_basic (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: 68 f7 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e h...........-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:58837 req_id = 29ead4bb-23e0-4795-add3-3e7e55cd283e
117992025-04-30T06:43:12.082ZDEBGtest_snapshot_basic (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 26871,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:58837 req_id = 29ead4bb-23e0-4795-add3-3e7e55cd283e
118002025-04-30T06:43:12.082ZDEBGtest_snapshot_basic (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
118012025-04-30T06:43:12.082ZDEBGtest_snapshot_basic (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:58837 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = 29ead4bb-23e0-4795-add3-3e7e55cd283e
118022025-04-30T06:43:12.138ZTRACtest_snapshot_basic (DnsResolver): lookup_all_socket_v6 srv dns_name = _clickhouse-admin-keeper._tcp.control-plane.oxide.internal
118032025-04-30T06:43:12.138ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
118042025-04-30T06:43:12.138ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {} }
118052025-04-30T06:43:12.140ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
118062025-04-30T06:43:12.140ZTRACtest_snapshot_basic (dns): buffer buffer = Length: 87 (0x57) bytes\n0000: 61 77 01 00 00 01 00 00 00 00 00 01 18 5f 63 6c aw..........._cl\n0010: 69 63 6b 68 6f 75 73 65 2d 61 64 6d 69 6e 2d 6b ickhouse-admin-k\n0020: 65 65 70 65 72 04 5f 74 63 70 0d 63 6f 6e 74 72 eeper._tcp.contr\n0030: 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 69 ol-plane.oxide.i\n0040: 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 04 nternal..!....).\n0050: d0 00 00 00 00 00 00 ....... kind = dns peer_addr = [::1]:53518 req_id = 8df9b5ad-9633-4e80-996b-16b224302c8d
118072025-04-30T06:43:12.140ZDEBGtest_snapshot_basic (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 24951,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 24,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 97,\n 100,\n 109,\n 105,\n 110,\n 45,\n 107,\n 101,\n 101,\n 112,\n 101,\n 114,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:53518 req_id = 8df9b5ad-9633-4e80-996b-16b224302c8d
118082025-04-30T06:43:12.141ZDEBGtest_snapshot_basic (store): query key key = _clickhouse-admin-keeper._tcp
118092025-04-30T06:43:12.141ZERROtest_snapshot_basic (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 24951, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } NXDOMAIN: no records found for name: "_clickhouse-admin-keeper._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:53518 req_id = 8df9b5ad-9633-4e80-996b-16b224302c8d
118102025-04-30T06:43:12.141ZWARNtest_snapshot_basic (results-sink-cluster): failed to insert some results into metric DB collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 error = Failed to check out connection to database: No backends found for this service file = oximeter/collector/src/results_sink.rs:92
118112025-04-30T06:43:12.142ZDEBGtest_snapshot_basic (BackgroundTasks): begin collection background_task = inventory_collection
118122025-04-30T06:43:12.142ZDEBGtest_snapshot_basic (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:41502
118132025-04-30T06:43:12.142ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/ignition
118142025-04-30T06:43:12.146ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "988eecbe-7783-4ad7-8d35-e8c36f8101c3", "content-length": "835", "date": "Wed, 30 Apr 2025 06:43:12 GMT"} })
118152025-04-30T06:43:12.147ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/0
118162025-04-30T06:43:12.154ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "b18e4858-8dc8-442c-a1dd-60c1785b4698", "content-length": "734", "date": "Wed, 30 Apr 2025 06:43:12 GMT"} })
118172025-04-30T06:43:12.165ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/0/component/sp/caboose?firmware_slot=0
118182025-04-30T06:43:12.170ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "d872ca83-8d7b-49cd-b743-568b0e042d15", "content-length": "111", "date": "Wed, 30 Apr 2025 06:43:12 GMT"} })
118192025-04-30T06:43:12.170ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/0/component/sp/caboose?firmware_slot=1
118202025-04-30T06:43:12.172ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "5965f3ad-ed50-4cef-bf4b-26770c65b869", "content-length": "111", "date": "Wed, 30 Apr 2025 06:43:12 GMT"} })
118212025-04-30T06:43:12.172ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/0/component/rot/caboose?firmware_slot=0
118222025-04-30T06:43:12.174ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "9d4a5ae8-9f56-48f6-8ea3-23e7bf977f14", "content-length": "170", "date": "Wed, 30 Apr 2025 06:43:12 GMT"} })
118232025-04-30T06:43:12.175ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/0/component/rot/caboose?firmware_slot=1
118242025-04-30T06:43:12.182ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "f80c0f05-5582-431d-b6a0-13fef494c0fd", "content-length": "170", "date": "Wed, 30 Apr 2025 06:43:12 GMT"} })
118252025-04-30T06:43:12.182ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/0/component/stage0/caboose?firmware_slot=0
118262025-04-30T06:43:12.186ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "c6687b2b-5702-4173-86f4-623bd8515244", "content-length": "179", "date": "Wed, 30 Apr 2025 06:43:12 GMT"} })
118272025-04-30T06:43:12.186ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/0/component/stage0/caboose?firmware_slot=1
118282025-04-30T06:43:12.187ZDEBGtest_snapshot_basic: attempt to fetch latest inventory collection result = Ok("found")
118292025-04-30T06:43:12.190ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "5657e22c-9081-4e2b-a8e1-57d14ad8d8df", "content-length": "179", "date": "Wed, 30 Apr 2025 06:43:12 GMT"} })
118302025-04-30T06:43:12.190ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/0/component/rot/cmpa
118312025-04-30T06:43:12.192ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "5c92b672-5ee1-41e7-8202-ce7bd57caf2c", "content-length": "702", "date": "Wed, 30 Apr 2025 06:43:12 GMT"} })
118322025-04-30T06:43:12.192ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/0/component/rot/cfpa
118332025-04-30T06:43:12.193ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "ef97edfe-9584-4aec-99be-fc2426998838", "content-length": "727", "date": "Wed, 30 Apr 2025 06:43:12 GMT"} })
118342025-04-30T06:43:12.194ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/0/component/rot/cfpa
118352025-04-30T06:43:12.196ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "14d69e08-ae32-4617-9c57-0ff295ec78ca", "content-length": "729", "date": "Wed, 30 Apr 2025 06:43:12 GMT"} })
118362025-04-30T06:43:12.196ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/0/component/rot/cfpa
118372025-04-30T06:43:12.197ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "a5b7327c-c10e-4a88-8e22-003dbac13e52", "content-length": "728", "date": "Wed, 30 Apr 2025 06:43:12 GMT"} })
118382025-04-30T06:43:12.197ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/1
118392025-04-30T06:43:12.198ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "c985a709-c657-4322-9466-b65647d035ca", "content-length": "734", "date": "Wed, 30 Apr 2025 06:43:12 GMT"} })
118402025-04-30T06:43:12.199ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/1/component/sp/caboose?firmware_slot=0
118412025-04-30T06:43:12.200ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "8ea1c8e7-b572-4e7e-8bc7-056083fc986a", "content-length": "111", "date": "Wed, 30 Apr 2025 06:43:12 GMT"} })
118422025-04-30T06:43:12.201ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/1/component/sp/caboose?firmware_slot=1
118432025-04-30T06:43:12.203ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "b937a75d-6be9-443b-8ae1-e65081579050", "content-length": "111", "date": "Wed, 30 Apr 2025 06:43:12 GMT"} })
118442025-04-30T06:43:12.203ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/1/component/rot/caboose?firmware_slot=0
118452025-04-30T06:43:12.208ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "fcda28d2-f3ed-4bdc-89a0-52e51ccef806", "content-length": "170", "date": "Wed, 30 Apr 2025 06:43:12 GMT"} })
118462025-04-30T06:43:12.208ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/1/component/rot/caboose?firmware_slot=1
118472025-04-30T06:43:12.211ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "5e29edab-fab4-479e-ab3f-a8ba0418e4d8", "content-length": "170", "date": "Wed, 30 Apr 2025 06:43:12 GMT"} })
118482025-04-30T06:43:12.211ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/1/component/stage0/caboose?firmware_slot=0
118492025-04-30T06:43:12.214ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "7705f78d-937d-4bfc-b57a-91244384e87f", "content-length": "179", "date": "Wed, 30 Apr 2025 06:43:12 GMT"} })
118502025-04-30T06:43:12.214ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/1/component/stage0/caboose?firmware_slot=1
118512025-04-30T06:43:12.216ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "05418e64-f5d4-49b8-8d3c-bfe075d72dbc", "content-length": "179", "date": "Wed, 30 Apr 2025 06:43:12 GMT"} })
118522025-04-30T06:43:12.216ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/1/component/rot/cmpa
118532025-04-30T06:43:12.218ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "d643fabc-8e91-499f-9083-5564472ceb97", "content-length": "702", "date": "Wed, 30 Apr 2025 06:43:12 GMT"} })
118542025-04-30T06:43:12.218ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/1/component/rot/cfpa
118552025-04-30T06:43:12.219ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "173d71db-a939-40a6-a3d9-5109a4786e57", "content-length": "727", "date": "Wed, 30 Apr 2025 06:43:12 GMT"} })
118562025-04-30T06:43:12.219ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/1/component/rot/cfpa
118572025-04-30T06:43:12.222ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "8dc4f662-dcd4-47c3-852c-020c8abdf2fc", "content-length": "729", "date": "Wed, 30 Apr 2025 06:43:12 GMT"} })
118582025-04-30T06:43:12.222ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/1/component/rot/cfpa
118592025-04-30T06:43:12.222ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "1390aeb1-2d58-49fa-a342-7e7172b5d59f", "content-length": "728", "date": "Wed, 30 Apr 2025 06:43:12 GMT"} })
118602025-04-30T06:43:12.222ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/0
118612025-04-30T06:43:12.224ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "9eb0e4c9-3471-4f2e-b309-b370bed30c29", "content-length": "723", "date": "Wed, 30 Apr 2025 06:43:12 GMT"} })
118622025-04-30T06:43:12.224ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/0/component/sp/caboose?firmware_slot=0
118632025-04-30T06:43:12.226ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "9460aa42-f3b7-4fd5-bb85-8ed3e2044ca1", "content-length": "109", "date": "Wed, 30 Apr 2025 06:43:12 GMT"} })
118642025-04-30T06:43:12.226ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/0/component/sp/caboose?firmware_slot=1
118652025-04-30T06:43:12.228ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "098b2c14-3fa6-497b-8c0a-81ed995025d6", "content-length": "109", "date": "Wed, 30 Apr 2025 06:43:12 GMT"} })
118662025-04-30T06:43:12.228ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/0/component/rot/caboose?firmware_slot=0
118672025-04-30T06:43:12.230ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "9d9281ba-f783-4588-8fbe-cfdbeed0a0a2", "content-length": "169", "date": "Wed, 30 Apr 2025 06:43:12 GMT"} })
118682025-04-30T06:43:12.233ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/0/component/rot/caboose?firmware_slot=1
118692025-04-30T06:43:12.235ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "7a1d7831-5e33-45f1-97f5-6329a8dabf40", "content-length": "169", "date": "Wed, 30 Apr 2025 06:43:12 GMT"} })
118702025-04-30T06:43:12.236ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/0/component/stage0/caboose?firmware_slot=0
118712025-04-30T06:43:12.237ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
118722025-04-30T06:43:12.238ZTRACtest_snapshot_basic (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
118732025-04-30T06:43:12.238ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
118742025-04-30T06:43:12.238ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {} }
118752025-04-30T06:43:12.238ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
118762025-04-30T06:43:12.239ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "f20c9a55-c32a-4344-9a85-bfde08ca81e1", "content-length": "178", "date": "Wed, 30 Apr 2025 06:43:12 GMT"} })
118772025-04-30T06:43:12.239ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/0/component/stage0/caboose?firmware_slot=1
118782025-04-30T06:43:12.241ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
118792025-04-30T06:43:12.251ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
118802025-04-30T06:43:12.251ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
118812025-04-30T06:43:12.251ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {} }
118822025-04-30T06:43:12.251ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
118832025-04-30T06:43:12.253ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
118842025-04-30T06:43:12.253ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {} }
118852025-04-30T06:43:12.254ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
118862025-04-30T06:43:12.254ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "22b1e85e-7040-404f-9032-4709e6b19b49", "content-length": "178", "date": "Wed, 30 Apr 2025 06:43:12 GMT"} })
118872025-04-30T06:43:12.254ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/0/component/rot/cmpa
118882025-04-30T06:43:12.255ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "e65df459-bf27-4ca1-a75a-4d75f725a3e4", "content-length": "702", "date": "Wed, 30 Apr 2025 06:43:12 GMT"} })
118892025-04-30T06:43:12.256ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/0/component/rot/cfpa
118902025-04-30T06:43:12.257ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "cebcf493-94b9-4072-af4e-9fb8bc55d3ac", "content-length": "727", "date": "Wed, 30 Apr 2025 06:43:12 GMT"} })
118912025-04-30T06:43:12.257ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/0/component/rot/cfpa
118922025-04-30T06:43:12.258ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "c8fc1676-ca91-4966-a433-e1136d38c283", "content-length": "729", "date": "Wed, 30 Apr 2025 06:43:12 GMT"} })
118932025-04-30T06:43:12.258ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/0/component/rot/cfpa
118942025-04-30T06:43:12.259ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "9ae299ce-5642-4daf-bd5a-bbbe02074661", "content-length": "728", "date": "Wed, 30 Apr 2025 06:43:12 GMT"} })
118952025-04-30T06:43:12.259ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/1
118962025-04-30T06:43:12.261ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "e5301ded-3771-412f-bf6f-98e2710b5b8c", "content-length": "723", "date": "Wed, 30 Apr 2025 06:43:12 GMT"} })
118972025-04-30T06:43:12.261ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/1/component/sp/caboose?firmware_slot=0
118982025-04-30T06:43:12.264ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "7b1010e2-b944-4868-839e-6b6b291c0e1b", "content-length": "109", "date": "Wed, 30 Apr 2025 06:43:12 GMT"} })
118992025-04-30T06:43:12.264ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/1/component/sp/caboose?firmware_slot=1
119002025-04-30T06:43:12.275ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "a2625cae-b986-4051-a669-9e2485a9156b", "content-length": "109", "date": "Wed, 30 Apr 2025 06:43:12 GMT"} })
119012025-04-30T06:43:12.276ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/1/component/rot/caboose?firmware_slot=0
119022025-04-30T06:43:12.279ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "1da66b82-be68-492a-bfab-93dc2a2927cc", "content-length": "169", "date": "Wed, 30 Apr 2025 06:43:12 GMT"} })
119032025-04-30T06:43:12.279ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/1/component/rot/caboose?firmware_slot=1
119042025-04-30T06:43:12.287ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "5f07d50c-1ecc-4c19-80d8-c4d22f2d8afc", "content-length": "169", "date": "Wed, 30 Apr 2025 06:43:12 GMT"} })
119052025-04-30T06:43:12.287ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/1/component/stage0/caboose?firmware_slot=0
119062025-04-30T06:43:12.291ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "4cee56d1-e61e-4289-b961-4f0d82b43ee7", "content-length": "178", "date": "Wed, 30 Apr 2025 06:43:12 GMT"} })
119072025-04-30T06:43:12.291ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/1/component/stage0/caboose?firmware_slot=1
119082025-04-30T06:43:12.296ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "d83b8c6b-eb45-4aad-967a-9a937b546de5", "content-length": "178", "date": "Wed, 30 Apr 2025 06:43:12 GMT"} })
119092025-04-30T06:43:12.296ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/1/component/rot/cmpa
119102025-04-30T06:43:12.297ZDEBGtest_snapshot_basic: attempt to fetch latest inventory collection result = Ok("found")
119112025-04-30T06:43:12.298ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "792b891c-83a7-444f-ab84-836362311fb2", "content-length": "702", "date": "Wed, 30 Apr 2025 06:43:12 GMT"} })
119122025-04-30T06:43:12.298ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/1/component/rot/cfpa
119132025-04-30T06:43:12.299ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "3a1ee06f-d411-4455-8da5-8b691ad47701", "content-length": "727", "date": "Wed, 30 Apr 2025 06:43:12 GMT"} })
119142025-04-30T06:43:12.299ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/1/component/rot/cfpa
119152025-04-30T06:43:12.301ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "2527b612-a194-450b-b340-fc6dd916fd39", "content-length": "729", "date": "Wed, 30 Apr 2025 06:43:12 GMT"} })
119162025-04-30T06:43:12.301ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/1/component/rot/cfpa
119172025-04-30T06:43:12.302ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "b7481108-4d62-4cbe-8b1f-e16bb5c7cdc1", "content-length": "728", "date": "Wed, 30 Apr 2025 06:43:12 GMT"} })
119182025-04-30T06:43:12.302ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
119192025-04-30T06:43:12.302ZTRACtest_snapshot_basic (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
119202025-04-30T06:43:12.303ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
119212025-04-30T06:43:12.303ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
119222025-04-30T06:43:12.305ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
119232025-04-30T06:43:12.308ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
119242025-04-30T06:43:12.314ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
119252025-04-30T06:43:12.314ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
119262025-04-30T06:43:12.314ZTRACtest_snapshot_basic (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
119272025-04-30T06:43:12.314ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
119282025-04-30T06:43:12.314ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
119292025-04-30T06:43:12.314ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
119302025-04-30T06:43:12.317ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
119312025-04-30T06:43:12.322ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
119322025-04-30T06:43:12.322ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
119332025-04-30T06:43:12.322ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
119342025-04-30T06:43:12.322ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
119352025-04-30T06:43:12.435ZDEBGtest_snapshot_basic (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:41923
119362025-04-30T06:43:12.435ZDEBGtest_snapshot_basic (BackgroundTasks): client request SledAgent = http://[::1]:41923 background_task = inventory_collection body = None method = GET uri = http://[::1]:41923/inventory
119372025-04-30T06:43:12.436ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
119382025-04-30T06:43:12.436ZTRACtest_snapshot_basic (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
119392025-04-30T06:43:12.436ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
119402025-04-30T06:43:12.436ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {} }
119412025-04-30T06:43:12.437ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
119422025-04-30T06:43:12.438ZINFOtest_snapshot_basic (dropshot): accepted connection local_addr = [::1]:41923 remote_addr = [::1]:61634 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
119432025-04-30T06:43:12.438ZTRACtest_snapshot_basic (dropshot): incoming request local_addr = [::1]:41923 method = GET remote_addr = [::1]:61634 req_id = 8cdab4a3-f300-409b-852a-2d5d32664a9f sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
119442025-04-30T06:43:12.439ZINFOtest_snapshot_basic (dropshot): request completed latency_us = 560 local_addr = [::1]:41923 method = GET remote_addr = [::1]:61634 req_id = 8cdab4a3-f300-409b-852a-2d5d32664a9f response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
119452025-04-30T06:43:12.439ZDEBGtest_snapshot_basic (BackgroundTasks): client response SledAgent = http://[::1]:41923 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:41923/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "8cdab4a3-f300-409b-852a-2d5d32664a9f", "content-length": "3141", "date": "Wed, 30 Apr 2025 06:43:12 GMT"} })
119462025-04-30T06:43:12.443ZDEBGtest_snapshot_basic (BackgroundTasks): begin collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
119472025-04-30T06:43:12.444ZDEBGtest_snapshot_basic (BackgroundTasks): end collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
119482025-04-30T06:43:12.444ZDEBGtest_snapshot_basic (BackgroundTasks): finished collection background_task = inventory_collection
119492025-04-30T06:43:12.447ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
119502025-04-30T06:43:12.447ZTRACtest_snapshot_basic (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
119512025-04-30T06:43:12.448ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
119522025-04-30T06:43:12.448ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
119532025-04-30T06:43:12.448ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
119542025-04-30T06:43:12.452ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
119552025-04-30T06:43:12.474ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
119562025-04-30T06:43:12.474ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
119572025-04-30T06:43:12.474ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {} }
119582025-04-30T06:43:12.474ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
119592025-04-30T06:43:12.476ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
119602025-04-30T06:43:12.481ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
119612025-04-30T06:43:12.482ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
119622025-04-30T06:43:12.482ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
119632025-04-30T06:43:12.483ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
119642025-04-30T06:43:12.490ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
119652025-04-30T06:43:12.490ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {} }
119662025-04-30T06:43:12.491ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
119672025-04-30T06:43:12.527ZWARNtest_snapshot_basic (datastore): Non-retryable transaction failure err = DatabaseError(UniqueViolation, "duplicate key value violates unique constraint \\"inv_caboose_pkey\\"") transaction = inventory_insert_collection
119682025-04-30T06:43:12.528ZWARNtest_snapshot_basic (BackgroundTasks): inventory collection failed background_task = inventory_collection error = failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint "inv_caboose_pkey"
119692025-04-30T06:43:12.528ZDEBGtest_snapshot_basic (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 518.707782ms iteration = 42 status = {"error":"failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint \\"inv_caboose_pkey\\""}
119702025-04-30T06:43:12.528ZDEBGtest_snapshot_basic (BackgroundTasks): activating background_task = inventory_collection iteration = 43 reason = Signaled
119712025-04-30T06:43:12.528ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
119722025-04-30T06:43:12.529ZTRACtest_snapshot_basic (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
119732025-04-30T06:43:12.529ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
119742025-04-30T06:43:12.529ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
119752025-04-30T06:43:12.531ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
119762025-04-30T06:43:12.536ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
119772025-04-30T06:43:12.541ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
119782025-04-30T06:43:12.542ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
119792025-04-30T06:43:12.542ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
119802025-04-30T06:43:12.543ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
119812025-04-30T06:43:12.548ZDEBGtest_snapshot_basic (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = [(d1ad44e5-18dd-4193-a70e-bf9042f0e0e1, 0)]
119822025-04-30T06:43:12.548ZTRACtest_snapshot_basic (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
119832025-04-30T06:43:12.559ZTRACtest_snapshot_basic (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: 4d 18 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 M............_mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:64919 req_id = 2a8fc55c-684c-459d-b9dc-d633a0409e62
119842025-04-30T06:43:12.561ZDEBGtest_snapshot_basic (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 19736,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:64919 req_id = 2a8fc55c-684c-459d-b9dc-d633a0409e62
119852025-04-30T06:43:12.561ZDEBGtest_snapshot_basic (store): query key key = _mgs._tcp
119862025-04-30T06:43:12.562ZDEBGtest_snapshot_basic (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
119872025-04-30T06:43:12.562ZDEBGtest_snapshot_basic (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:64919 query = LowerQuery { name: LowerName(Name("_mgs._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 41502, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = 2a8fc55c-684c-459d-b9dc-d633a0409e62
119882025-04-30T06:43:12.567ZDEBGtest_snapshot_basic (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 41502, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 4257, tv_nsec: 366471995 } })
119892025-04-30T06:43:12.567ZTRACtest_snapshot_basic (DnsResolver): lookup_service_targets: looking up SRV target
119902025-04-30T06:43:12.569ZDEBGtest_snapshot_basic: attempt to fetch latest inventory collection result = Ok("found")
119912025-04-30T06:43:12.576ZTRACtest_snapshot_basic (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: d5 f3 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e ............-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:49509 req_id = 73fb09ab-1523-4528-9137-06702ff7f2f8
119922025-04-30T06:43:12.578ZDEBGtest_snapshot_basic (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 54771,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:49509 req_id = 73fb09ab-1523-4528-9137-06702ff7f2f8
119932025-04-30T06:43:12.578ZDEBGtest_snapshot_basic (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
119942025-04-30T06:43:12.579ZDEBGtest_snapshot_basic (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:49509 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = 73fb09ab-1523-4528-9137-06702ff7f2f8
119952025-04-30T06:43:12.671ZTRACtest_snapshot_basic (DnsResolver): lookup_all_socket_v6 srv dns_name = _clickhouse-admin-keeper._tcp.control-plane.oxide.internal
119962025-04-30T06:43:12.675ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
119972025-04-30T06:43:12.675ZTRACtest_snapshot_basic (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
119982025-04-30T06:43:12.675ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
119992025-04-30T06:43:12.675ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {} }
120002025-04-30T06:43:12.677ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
120012025-04-30T06:43:12.678ZTRACtest_snapshot_basic (dns): buffer buffer = Length: 87 (0x57) bytes\n0000: 34 a1 01 00 00 01 00 00 00 00 00 01 18 5f 63 6c 4............_cl\n0010: 69 63 6b 68 6f 75 73 65 2d 61 64 6d 69 6e 2d 6b ickhouse-admin-k\n0020: 65 65 70 65 72 04 5f 74 63 70 0d 63 6f 6e 74 72 eeper._tcp.contr\n0030: 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 69 ol-plane.oxide.i\n0040: 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 04 nternal..!....).\n0050: d0 00 00 00 00 00 00 ....... kind = dns peer_addr = [::1]:65504 req_id = 2a5821a7-f1fa-49c0-8553-2f432a4b4508
120022025-04-30T06:43:12.678ZDEBGtest_snapshot_basic (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 13473,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 24,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 97,\n 100,\n 109,\n 105,\n 110,\n 45,\n 107,\n 101,\n 101,\n 112,\n 101,\n 114,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:65504 req_id = 2a5821a7-f1fa-49c0-8553-2f432a4b4508
120032025-04-30T06:43:12.678ZDEBGtest_snapshot_basic (store): query key key = _clickhouse-admin-keeper._tcp
120042025-04-30T06:43:12.678ZERROtest_snapshot_basic (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 13473, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } NXDOMAIN: no records found for name: "_clickhouse-admin-keeper._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:65504 req_id = 2a5821a7-f1fa-49c0-8553-2f432a4b4508
120052025-04-30T06:43:12.678ZDEBGtest_snapshot_basic (BackgroundTasks): begin collection background_task = inventory_collection
120062025-04-30T06:43:12.679ZDEBGtest_snapshot_basic (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:41502
120072025-04-30T06:43:12.679ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/ignition
120082025-04-30T06:43:12.680ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
120092025-04-30T06:43:12.687ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
120102025-04-30T06:43:12.687ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
120112025-04-30T06:43:12.687ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {} }
120122025-04-30T06:43:12.687ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
120132025-04-30T06:43:12.689ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "f88b8a1e-ca12-47fb-b7f6-d8abe72272d5", "content-length": "835", "date": "Wed, 30 Apr 2025 06:43:12 GMT"} })
120142025-04-30T06:43:12.690ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/0
120152025-04-30T06:43:12.691ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
120162025-04-30T06:43:12.691ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {} }
120172025-04-30T06:43:12.691ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
120182025-04-30T06:43:12.692ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "228c8dd0-9e5d-4faa-b55f-79dab311790d", "content-length": "734", "date": "Wed, 30 Apr 2025 06:43:12 GMT"} })
120192025-04-30T06:43:12.692ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/0/component/sp/caboose?firmware_slot=0
120202025-04-30T06:43:12.696ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "6cc5f4c7-90ac-4ee7-9ec8-59e6f05971ee", "content-length": "111", "date": "Wed, 30 Apr 2025 06:43:12 GMT"} })
120212025-04-30T06:43:12.696ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/0/component/sp/caboose?firmware_slot=1
120222025-04-30T06:43:12.698ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "ca8a3473-7fda-41a6-80ea-41565594e0f0", "content-length": "111", "date": "Wed, 30 Apr 2025 06:43:12 GMT"} })
120232025-04-30T06:43:12.698ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/0/component/rot/caboose?firmware_slot=0
120242025-04-30T06:43:12.701ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "c15f857a-a701-46ec-b112-aeee1769e999", "content-length": "170", "date": "Wed, 30 Apr 2025 06:43:12 GMT"} })
120252025-04-30T06:43:12.701ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/0/component/rot/caboose?firmware_slot=1
120262025-04-30T06:43:12.704ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "c82c0042-463b-44f1-85b8-098946115648", "content-length": "170", "date": "Wed, 30 Apr 2025 06:43:12 GMT"} })
120272025-04-30T06:43:12.704ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/0/component/stage0/caboose?firmware_slot=0
120282025-04-30T06:43:12.707ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "322aef22-07a1-42a9-bc6c-438d16046991", "content-length": "179", "date": "Wed, 30 Apr 2025 06:43:12 GMT"} })
120292025-04-30T06:43:12.707ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/0/component/stage0/caboose?firmware_slot=1
120302025-04-30T06:43:12.710ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "3f7c8d3a-c6e9-4ce7-baf0-25864534163d", "content-length": "179", "date": "Wed, 30 Apr 2025 06:43:12 GMT"} })
120312025-04-30T06:43:12.711ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/0/component/rot/cmpa
120322025-04-30T06:43:12.712ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "6f9edf8a-8179-4778-9eaf-3af3e2116918", "content-length": "702", "date": "Wed, 30 Apr 2025 06:43:12 GMT"} })
120332025-04-30T06:43:12.713ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/0/component/rot/cfpa
120342025-04-30T06:43:12.714ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "e26c9e47-1868-4b21-86be-64ef7a91a668", "content-length": "727", "date": "Wed, 30 Apr 2025 06:43:12 GMT"} })
120352025-04-30T06:43:12.714ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/0/component/rot/cfpa
120362025-04-30T06:43:12.715ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "9f6dbd13-a38b-47a8-9724-4b594de60a79", "content-length": "729", "date": "Wed, 30 Apr 2025 06:43:12 GMT"} })
120372025-04-30T06:43:12.715ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/0/component/rot/cfpa
120382025-04-30T06:43:12.716ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "953e7132-d0a6-46eb-9185-41483314d110", "content-length": "728", "date": "Wed, 30 Apr 2025 06:43:12 GMT"} })
120392025-04-30T06:43:12.716ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/1
120402025-04-30T06:43:12.719ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "2c921462-d57a-4397-8adb-10a32e4181b2", "content-length": "734", "date": "Wed, 30 Apr 2025 06:43:12 GMT"} })
120412025-04-30T06:43:12.719ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/1/component/sp/caboose?firmware_slot=0
120422025-04-30T06:43:12.720ZDEBGtest_snapshot_basic: attempt to fetch latest inventory collection result = Ok("found")
120432025-04-30T06:43:12.722ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "0e785f75-bb9a-499e-9eb5-ece3b7035cfc", "content-length": "111", "date": "Wed, 30 Apr 2025 06:43:12 GMT"} })
120442025-04-30T06:43:12.723ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/1/component/sp/caboose?firmware_slot=1
120452025-04-30T06:43:12.725ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "88a0450f-9845-4d95-8203-faacb8cc172d", "content-length": "111", "date": "Wed, 30 Apr 2025 06:43:12 GMT"} })
120462025-04-30T06:43:12.725ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/1/component/rot/caboose?firmware_slot=0
120472025-04-30T06:43:12.727ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "dd3ccede-76f8-472a-a46d-8cd8b3f5ac2c", "content-length": "170", "date": "Wed, 30 Apr 2025 06:43:12 GMT"} })
120482025-04-30T06:43:12.727ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/1/component/rot/caboose?firmware_slot=1
120492025-04-30T06:43:12.729ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "3e272873-9618-4601-ac88-78c359d5df8d", "content-length": "170", "date": "Wed, 30 Apr 2025 06:43:12 GMT"} })
120502025-04-30T06:43:12.729ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/1/component/stage0/caboose?firmware_slot=0
120512025-04-30T06:43:12.732ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "ac0799f4-c5a0-404b-baa0-849852d923e4", "content-length": "179", "date": "Wed, 30 Apr 2025 06:43:12 GMT"} })
120522025-04-30T06:43:12.732ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/1/component/stage0/caboose?firmware_slot=1
120532025-04-30T06:43:12.736ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "60e84b65-517d-4e14-bbad-624c23da5750", "content-length": "179", "date": "Wed, 30 Apr 2025 06:43:12 GMT"} })
120542025-04-30T06:43:12.736ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/1/component/rot/cmpa
120552025-04-30T06:43:12.737ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "6e1409bd-057e-4026-bfb9-001675716039", "content-length": "702", "date": "Wed, 30 Apr 2025 06:43:12 GMT"} })
120562025-04-30T06:43:12.738ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/1/component/rot/cfpa
120572025-04-30T06:43:12.739ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "4e17bccb-315c-425a-8d42-12184fc064c1", "content-length": "727", "date": "Wed, 30 Apr 2025 06:43:12 GMT"} })
120582025-04-30T06:43:12.739ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/1/component/rot/cfpa
120592025-04-30T06:43:12.740ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "c249bba9-17fa-4099-9750-f5f797f8f42c", "content-length": "729", "date": "Wed, 30 Apr 2025 06:43:12 GMT"} })
120602025-04-30T06:43:12.740ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/1/component/rot/cfpa
120612025-04-30T06:43:12.742ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "624495f4-5f9e-4d94-9d2e-84f67e238dad", "content-length": "728", "date": "Wed, 30 Apr 2025 06:43:12 GMT"} })
120622025-04-30T06:43:12.742ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/0
120632025-04-30T06:43:12.743ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "36530712-f244-4405-b37b-70db9ebcdf65", "content-length": "723", "date": "Wed, 30 Apr 2025 06:43:12 GMT"} })
120642025-04-30T06:43:12.744ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/0/component/sp/caboose?firmware_slot=0
120652025-04-30T06:43:12.746ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "b66c8035-e7a9-41dc-af74-6a474d721b43", "content-length": "109", "date": "Wed, 30 Apr 2025 06:43:12 GMT"} })
120662025-04-30T06:43:12.746ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/0/component/sp/caboose?firmware_slot=1
120672025-04-30T06:43:12.749ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "d04837b4-b8e1-4899-a729-5ffa8c41edb7", "content-length": "109", "date": "Wed, 30 Apr 2025 06:43:12 GMT"} })
120682025-04-30T06:43:12.749ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/0/component/rot/caboose?firmware_slot=0
120692025-04-30T06:43:12.755ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "8d26d8d3-84a4-44be-b90d-57e6cc2f42c6", "content-length": "169", "date": "Wed, 30 Apr 2025 06:43:12 GMT"} })
120702025-04-30T06:43:12.755ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/0/component/rot/caboose?firmware_slot=1
120712025-04-30T06:43:12.758ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "fe7775ac-9641-40f3-bb38-166bcdaa5277", "content-length": "169", "date": "Wed, 30 Apr 2025 06:43:12 GMT"} })
120722025-04-30T06:43:12.758ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/0/component/stage0/caboose?firmware_slot=0
120732025-04-30T06:43:12.763ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "21a2d54f-b985-4ccf-8055-6825fc0c2feb", "content-length": "178", "date": "Wed, 30 Apr 2025 06:43:12 GMT"} })
120742025-04-30T06:43:12.763ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/0/component/stage0/caboose?firmware_slot=1
120752025-04-30T06:43:12.767ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "a6134096-690f-4ce0-98fb-20ce7f6b298f", "content-length": "178", "date": "Wed, 30 Apr 2025 06:43:12 GMT"} })
120762025-04-30T06:43:12.767ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/0/component/rot/cmpa
120772025-04-30T06:43:12.768ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "a6d4eb3e-a145-47b5-bc1f-9bbc8536c7bf", "content-length": "702", "date": "Wed, 30 Apr 2025 06:43:12 GMT"} })
120782025-04-30T06:43:12.769ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/0/component/rot/cfpa
120792025-04-30T06:43:12.770ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "223cce17-c32e-4716-944c-b9f78ae30a6c", "content-length": "727", "date": "Wed, 30 Apr 2025 06:43:12 GMT"} })
120802025-04-30T06:43:12.770ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/0/component/rot/cfpa
120812025-04-30T06:43:12.771ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
120822025-04-30T06:43:12.771ZTRACtest_snapshot_basic (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
120832025-04-30T06:43:12.771ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
120842025-04-30T06:43:12.771ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {} }
120852025-04-30T06:43:12.772ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
120862025-04-30T06:43:12.773ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "130ef209-9841-4baf-a576-7ec505716679", "content-length": "729", "date": "Wed, 30 Apr 2025 06:43:12 GMT"} })
120872025-04-30T06:43:12.773ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/0/component/rot/cfpa
120882025-04-30T06:43:12.778ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
120892025-04-30T06:43:12.786ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
120902025-04-30T06:43:12.786ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
120912025-04-30T06:43:12.786ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {} }
120922025-04-30T06:43:12.787ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
120932025-04-30T06:43:12.791ZTRACtest_snapshot_basic (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal
120942025-04-30T06:43:12.792ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "70846228-afe8-44cc-8011-9bd0312aa585", "content-length": "728", "date": "Wed, 30 Apr 2025 06:43:12 GMT"} })
120952025-04-30T06:43:12.792ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/1
120962025-04-30T06:43:12.793ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
120972025-04-30T06:43:12.795ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {} }
120982025-04-30T06:43:12.796ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
120992025-04-30T06:43:12.800ZDEBGtest_snapshot_basic (collection-task): sent timer-based collection request to the collection task collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
121002025-04-30T06:43:12.800ZTRACtest_snapshot_basic (dns): buffer buffer = Length: 72 (0x48) bytes\n0000: cc 2b 01 00 00 01 00 00 00 00 00 01 09 5f 64 65 .+..........._de\n0010: 6e 64 72 69 74 65 04 5f 74 63 70 0d 63 6f 6e 74 ndrite._tcp.cont\n0020: 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 rol-plane.oxide.\n0030: 69 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 internal..!....)\n0040: 04 d0 00 00 00 00 00 00 ........ kind = dns peer_addr = [::1]:50204 req_id = 96795bf3-5d0a-431b-88ee-85c2ba67d33f
121012025-04-30T06:43:12.800ZDEBGtest_snapshot_basic (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 52267,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_dendrite._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_dendrite._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 9,\n 95,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:50204 req_id = 96795bf3-5d0a-431b-88ee-85c2ba67d33f
121022025-04-30T06:43:12.801ZDEBGtest_snapshot_basic (store): query key key = _dendrite._tcp
121032025-04-30T06:43:12.801ZDEBGtest_snapshot_basic (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
121042025-04-30T06:43:12.801ZDEBGtest_snapshot_basic (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:50204 query = LowerQuery { name: LowerName(Name("_dendrite._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 32976, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = 96795bf3-5d0a-431b-88ee-85c2ba67d33f
121052025-04-30T06:43:12.801ZDEBGtest_snapshot_basic (collection-task): collecting from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
121062025-04-30T06:43:12.802ZDEBGtest_snapshot_basic (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 32976, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 4257, tv_nsec: 601445784 } })
121072025-04-30T06:43:12.802ZTRACtest_snapshot_basic (DnsResolver): lookup_service_targets: looking up SRV target
121082025-04-30T06:43:12.806ZTRACtest_snapshot_basic (collection-task): sent collection request to producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
121092025-04-30T06:43:12.807ZDEBGtest_snapshot_basic (collection-task): collected results from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 n_results = 1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
121102025-04-30T06:43:12.807ZTRACtest_snapshot_basic (collection-task): forwarded results to main collection loop collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
121112025-04-30T06:43:12.807ZTRACtest_snapshot_basic (collection-task): top of inner collection loop, waiting for next request collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
121122025-04-30T06:43:12.808ZTRACtest_snapshot_basic (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: d0 66 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e .f..........-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:57631 req_id = 374c5744-fea0-4bcf-9b78-d8ca298e33a1
121132025-04-30T06:43:12.808ZDEBGtest_snapshot_basic (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 53350,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:57631 req_id = 374c5744-fea0-4bcf-9b78-d8ca298e33a1
121142025-04-30T06:43:12.808ZDEBGtest_snapshot_basic (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
121152025-04-30T06:43:12.808ZDEBGtest_snapshot_basic (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:57631 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = 374c5744-fea0-4bcf-9b78-d8ca298e33a1
121162025-04-30T06:43:12.810ZINFOtest_snapshot_basic (BackgroundTasks): Determining switch slots managed by switch zones background_task = nat_v4_garbage_collector
121172025-04-30T06:43:12.879ZINFOtest_snapshot_basic (BackgroundTasks): determining switch slot managed by dendrite zone background_task = nat_v4_garbage_collector zone_address = ::1
121182025-04-30T06:43:12.880ZDEBGtest_snapshot_basic (MgsClient): client request background_task = nat_v4_garbage_collector body = None method = GET uri = http://[::1]:12225/local/switch-id
121192025-04-30T06:43:12.880ZDEBGtest_snapshot_basic (MgsClient): client response background_task = nat_v4_garbage_collector result = Err(reqwest::Error { kind: Request, url: "http://[::1]:12225/local/switch-id", source: hyper_util::client::legacy::Error(Connect, ConnectError("tcp connect error", Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) })
121202025-04-30T06:43:12.880ZWARNtest_snapshot_basic (BackgroundTasks): failed to identify switch slot for dendrite background_task = nat_v4_garbage_collector reason = Communication Error: error sending request for url (http://[::1]:12225/local/switch-id): client error (Connect): tcp connect error: Connection refused (os error 146): Connection refused (os error 146) zone_address = ::1
121212025-04-30T06:43:12.882ZWARNtest_snapshot_basic (BackgroundTasks): Failed to map switch zone addr: Communication Error: error sending request for url (http://[::1]:12225/local/switch-id), retrying background_task = nat_v4_garbage_collector
121222025-04-30T06:43:12.882ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "9c8cedaf-1b99-481a-b418-e9a5bd541003", "content-length": "723", "date": "Wed, 30 Apr 2025 06:43:12 GMT"} })
121232025-04-30T06:43:12.883ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/1/component/sp/caboose?firmware_slot=0
121242025-04-30T06:43:12.884ZTRACtest_snapshot_basic (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal
121252025-04-30T06:43:12.884ZTRACtest_snapshot_basic (dns): buffer buffer = Length: 72 (0x48) bytes\n0000: b3 b5 01 00 00 01 00 00 00 00 00 01 09 5f 64 65 ............._de\n0010: 6e 64 72 69 74 65 04 5f 74 63 70 0d 63 6f 6e 74 ndrite._tcp.cont\n0020: 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 rol-plane.oxide.\n0030: 69 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 internal..!....)\n0040: 04 d0 00 00 00 00 00 00 ........ kind = dns peer_addr = [::1]:49224 req_id = 960080ec-a041-4ff6-a871-e0e6b4ce5851
121262025-04-30T06:43:12.885ZDEBGtest_snapshot_basic (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 46005,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_dendrite._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_dendrite._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 9,\n 95,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:49224 req_id = 960080ec-a041-4ff6-a871-e0e6b4ce5851
121272025-04-30T06:43:12.885ZDEBGtest_snapshot_basic (store): query key key = _dendrite._tcp
121282025-04-30T06:43:12.885ZDEBGtest_snapshot_basic (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
121292025-04-30T06:43:12.886ZDEBGtest_snapshot_basic (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:49224 query = LowerQuery { name: LowerName(Name("_dendrite._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 32976, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = 960080ec-a041-4ff6-a871-e0e6b4ce5851
121302025-04-30T06:43:12.887ZDEBGtest_snapshot_basic (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 32976, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 4257, tv_nsec: 686047487 } })
121312025-04-30T06:43:12.887ZTRACtest_snapshot_basic (DnsResolver): lookup_service_targets: looking up SRV target
121322025-04-30T06:43:12.887ZTRACtest_snapshot_basic (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: 0d 91 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e ............-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:50372 req_id = ff52a2dc-f421-4757-a284-4d0fb1fbf55d
121332025-04-30T06:43:12.888ZDEBGtest_snapshot_basic (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 3473,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:50372 req_id = ff52a2dc-f421-4757-a284-4d0fb1fbf55d
121342025-04-30T06:43:12.889ZDEBGtest_snapshot_basic (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
121352025-04-30T06:43:12.889ZDEBGtest_snapshot_basic (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:50372 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = ff52a2dc-f421-4757-a284-4d0fb1fbf55d
121362025-04-30T06:43:12.890ZINFOtest_snapshot_basic (BackgroundTasks): Determining switch slots managed by switch zones background_task = bfd_manager
121372025-04-30T06:43:12.934ZINFOtest_snapshot_basic (BackgroundTasks): determining switch slot managed by dendrite zone background_task = bfd_manager zone_address = ::1
121382025-04-30T06:43:12.935ZDEBGtest_snapshot_basic (MgsClient): client request background_task = bfd_manager body = None method = GET uri = http://[::1]:12225/local/switch-id
121392025-04-30T06:43:12.935ZDEBGtest_snapshot_basic (MgsClient): client response background_task = bfd_manager result = Err(reqwest::Error { kind: Request, url: "http://[::1]:12225/local/switch-id", source: hyper_util::client::legacy::Error(Connect, ConnectError("tcp connect error", Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) })
121402025-04-30T06:43:12.935ZWARNtest_snapshot_basic (BackgroundTasks): failed to identify switch slot for dendrite background_task = bfd_manager reason = Communication Error: error sending request for url (http://[::1]:12225/local/switch-id): client error (Connect): tcp connect error: Connection refused (os error 146): Connection refused (os error 146) zone_address = ::1
121412025-04-30T06:43:12.937ZWARNtest_snapshot_basic (BackgroundTasks): Failed to map switch zone addr: Communication Error: error sending request for url (http://[::1]:12225/local/switch-id), retrying background_task = bfd_manager
121422025-04-30T06:43:12.940ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "203f3af9-402e-4bcb-8f3f-1a7c47fab0cb", "content-length": "109", "date": "Wed, 30 Apr 2025 06:43:12 GMT"} })
121432025-04-30T06:43:12.940ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/1/component/sp/caboose?firmware_slot=1
121442025-04-30T06:43:12.946ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "d1289bc9-20d4-45a9-8959-8b4d387eac5d", "content-length": "109", "date": "Wed, 30 Apr 2025 06:43:12 GMT"} })
121452025-04-30T06:43:12.946ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/1/component/rot/caboose?firmware_slot=0
121462025-04-30T06:43:12.948ZTRACtest_snapshot_basic (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal
121472025-04-30T06:43:12.949ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "979be9f6-95b0-47e9-94ee-cf0ea9ed9897", "content-length": "169", "date": "Wed, 30 Apr 2025 06:43:12 GMT"} })
121482025-04-30T06:43:12.949ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/1/component/rot/caboose?firmware_slot=1
121492025-04-30T06:43:12.950ZTRACtest_snapshot_basic (dns): buffer buffer = Length: 72 (0x48) bytes\n0000: d9 51 01 00 00 01 00 00 00 00 00 01 09 5f 64 65 .Q..........._de\n0010: 6e 64 72 69 74 65 04 5f 74 63 70 0d 63 6f 6e 74 ndrite._tcp.cont\n0020: 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 rol-plane.oxide.\n0030: 69 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 internal..!....)\n0040: 04 d0 00 00 00 00 00 00 ........ kind = dns peer_addr = [::1]:60138 req_id = 441163bb-4378-40c0-a171-ade2e07ca6c7
121502025-04-30T06:43:12.950ZDEBGtest_snapshot_basic (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 55633,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_dendrite._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_dendrite._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 9,\n 95,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:60138 req_id = 441163bb-4378-40c0-a171-ade2e07ca6c7
121512025-04-30T06:43:12.950ZDEBGtest_snapshot_basic (store): query key key = _dendrite._tcp
121522025-04-30T06:43:12.950ZDEBGtest_snapshot_basic (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
121532025-04-30T06:43:12.951ZDEBGtest_snapshot_basic (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:60138 query = LowerQuery { name: LowerName(Name("_dendrite._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 32976, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = 441163bb-4378-40c0-a171-ade2e07ca6c7
121542025-04-30T06:43:12.951ZDEBGtest_snapshot_basic (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 32976, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 4257, tv_nsec: 750636527 } })
121552025-04-30T06:43:12.951ZTRACtest_snapshot_basic (DnsResolver): lookup_service_targets: looking up SRV target
121562025-04-30T06:43:12.952ZTRACtest_snapshot_basic (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: e1 92 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e ............-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:61172 req_id = cae7e846-8687-4a84-99ce-81f857dc594b
121572025-04-30T06:43:12.952ZDEBGtest_snapshot_basic (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 57746,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:61172 req_id = cae7e846-8687-4a84-99ce-81f857dc594b
121582025-04-30T06:43:12.952ZDEBGtest_snapshot_basic (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
121592025-04-30T06:43:12.952ZDEBGtest_snapshot_basic (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:61172 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = cae7e846-8687-4a84-99ce-81f857dc594b
121602025-04-30T06:43:12.953ZINFOtest_snapshot_basic (BackgroundTasks): Determining switch slots managed by switch zones background_task = switch_port_config_manager rack_id = c19a698f-c6f9-4a17-ae30-20d711b8f7dc
121612025-04-30T06:43:13.018ZINFOtest_snapshot_basic (BackgroundTasks): determining switch slot managed by dendrite zone background_task = switch_port_config_manager rack_id = c19a698f-c6f9-4a17-ae30-20d711b8f7dc zone_address = ::1
121622025-04-30T06:43:13.018ZDEBGtest_snapshot_basic (MgsClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = c19a698f-c6f9-4a17-ae30-20d711b8f7dc uri = http://[::1]:12225/local/switch-id
121632025-04-30T06:43:13.018ZDEBGtest_snapshot_basic (MgsClient): client response background_task = switch_port_config_manager rack_id = c19a698f-c6f9-4a17-ae30-20d711b8f7dc result = Err(reqwest::Error { kind: Request, url: "http://[::1]:12225/local/switch-id", source: hyper_util::client::legacy::Error(Connect, ConnectError("tcp connect error", Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) })
121642025-04-30T06:43:13.018ZWARNtest_snapshot_basic (BackgroundTasks): failed to identify switch slot for dendrite background_task = switch_port_config_manager rack_id = c19a698f-c6f9-4a17-ae30-20d711b8f7dc reason = Communication Error: error sending request for url (http://[::1]:12225/local/switch-id): client error (Connect): tcp connect error: Connection refused (os error 146): Connection refused (os error 146) zone_address = ::1
121652025-04-30T06:43:13.022ZWARNtest_snapshot_basic (BackgroundTasks): Failed to map switch zone addr: Communication Error: error sending request for url (http://[::1]:12225/local/switch-id), retrying background_task = switch_port_config_manager rack_id = c19a698f-c6f9-4a17-ae30-20d711b8f7dc
121662025-04-30T06:43:13.024ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "fb43e5a1-b2bb-4442-9d2d-4c9649d81ab3", "content-length": "169", "date": "Wed, 30 Apr 2025 06:43:12 GMT"} })
121672025-04-30T06:43:13.024ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/1/component/stage0/caboose?firmware_slot=0
121682025-04-30T06:43:13.027ZDEBGtest_snapshot_basic: attempt to fetch latest inventory collection result = Ok("found")
121692025-04-30T06:43:13.034ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "15d821b6-8edc-4caa-a4af-154afeea20a7", "content-length": "178", "date": "Wed, 30 Apr 2025 06:43:12 GMT"} })
121702025-04-30T06:43:13.036ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/1/component/stage0/caboose?firmware_slot=1
121712025-04-30T06:43:13.036ZDEBGtest_snapshot_basic (results-sink): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
121722025-04-30T06:43:13.037ZDEBGtest_snapshot_basic (results-sink-cluster): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
121732025-04-30T06:43:13.037ZDEBGtest_snapshot_basic (clickhouse-client): unrolling 1 total samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 6ad06c6b-9a6e-4ed1-b3f4-5d2e9da3f8a7
121742025-04-30T06:43:13.039ZTRACtest_snapshot_basic (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 6ad06c6b-9a6e-4ed1-b3f4-5d2e9da3f8a7 n_columns = 4 n_rows = 2 sql = INSERT INTO oximeter.fields_string FORMAT Native
121752025-04-30T06:43:13.043ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "3ae37fd0-fddd-4c98-ad27-d8e5a0bb7c55", "content-length": "178", "date": "Wed, 30 Apr 2025 06:43:12 GMT"} })
121762025-04-30T06:43:13.043ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/1/component/rot/cmpa
121772025-04-30T06:43:13.044ZDEBGtest_snapshot_basic (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 6ad06c6b-9a6e-4ed1-b3f4-5d2e9da3f8a7 n_rows = 2 table_name = fields_string
121782025-04-30T06:43:13.044ZTRACtest_snapshot_basic (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 6ad06c6b-9a6e-4ed1-b3f4-5d2e9da3f8a7 n_columns = 4 n_rows = 1 sql = INSERT INTO oximeter.measurements_i64 FORMAT Native
121792025-04-30T06:43:13.045ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "72eb8b79-173f-46a1-9b4d-9091ed23a10f", "content-length": "702", "date": "Wed, 30 Apr 2025 06:43:12 GMT"} })
121802025-04-30T06:43:13.045ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/1/component/rot/cfpa
121812025-04-30T06:43:13.048ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "d3ba2e77-61cc-4689-8258-c68f37f993f8", "content-length": "727", "date": "Wed, 30 Apr 2025 06:43:12 GMT"} })
121822025-04-30T06:43:13.048ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/1/component/rot/cfpa
121832025-04-30T06:43:13.053ZDEBGtest_snapshot_basic (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 6ad06c6b-9a6e-4ed1-b3f4-5d2e9da3f8a7 n_rows = 1 table_name = measurements_i64
121842025-04-30T06:43:13.053ZTRACtest_snapshot_basic (results-sink): successfully inserted samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
121852025-04-30T06:43:13.053ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "b5d9d872-d83f-421d-a348-490036b1e688", "content-length": "729", "date": "Wed, 30 Apr 2025 06:43:12 GMT"} })
121862025-04-30T06:43:13.053ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/1/component/rot/cfpa
121872025-04-30T06:43:13.056ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "1a444a5d-d104-44fe-a770-ccfcb625b9aa", "content-length": "728", "date": "Wed, 30 Apr 2025 06:43:12 GMT"} })
121882025-04-30T06:43:13.056ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
121892025-04-30T06:43:13.056ZTRACtest_snapshot_basic (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
121902025-04-30T06:43:13.056ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
121912025-04-30T06:43:13.056ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
121922025-04-30T06:43:13.057ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
121932025-04-30T06:43:13.060ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
121942025-04-30T06:43:13.067ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
121952025-04-30T06:43:13.067ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
121962025-04-30T06:43:13.067ZTRACtest_snapshot_basic (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
121972025-04-30T06:43:13.067ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
121982025-04-30T06:43:13.067ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
121992025-04-30T06:43:13.068ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
122002025-04-30T06:43:13.076ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
122012025-04-30T06:43:13.082ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
122022025-04-30T06:43:13.083ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
122032025-04-30T06:43:13.083ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
122042025-04-30T06:43:13.084ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
122052025-04-30T06:43:13.085ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
122062025-04-30T06:43:13.085ZTRACtest_snapshot_basic (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
122072025-04-30T06:43:13.086ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
122082025-04-30T06:43:13.086ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {} }
122092025-04-30T06:43:13.087ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
122102025-04-30T06:43:13.091ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
122112025-04-30T06:43:13.096ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
122122025-04-30T06:43:13.097ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
122132025-04-30T06:43:13.097ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {} }
122142025-04-30T06:43:13.097ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
122152025-04-30T06:43:13.160ZDEBGtest_snapshot_basic (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:41923
122162025-04-30T06:43:13.161ZDEBGtest_snapshot_basic (BackgroundTasks): client request SledAgent = http://[::1]:41923 background_task = inventory_collection body = None method = GET uri = http://[::1]:41923/inventory
122172025-04-30T06:43:13.165ZINFOtest_snapshot_basic (dropshot): accepted connection local_addr = [::1]:41923 remote_addr = [::1]:40262 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
122182025-04-30T06:43:13.166ZWARNtest_snapshot_basic (results-sink-cluster): failed to insert some results into metric DB collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 error = Failed to check out connection to database: No backends found for this service file = oximeter/collector/src/results_sink.rs:92
122192025-04-30T06:43:13.167ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
122202025-04-30T06:43:13.167ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {} }
122212025-04-30T06:43:13.169ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
122222025-04-30T06:43:13.170ZTRACtest_snapshot_basic (dropshot): incoming request local_addr = [::1]:41923 method = GET remote_addr = [::1]:40262 req_id = c98401f8-6992-4d51-829c-9a0e1fa0509e sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
122232025-04-30T06:43:13.171ZINFOtest_snapshot_basic (dropshot): request completed latency_us = 1603 local_addr = [::1]:41923 method = GET remote_addr = [::1]:40262 req_id = c98401f8-6992-4d51-829c-9a0e1fa0509e response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
122242025-04-30T06:43:13.172ZDEBGtest_snapshot_basic (BackgroundTasks): client response SledAgent = http://[::1]:41923 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:41923/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "c98401f8-6992-4d51-829c-9a0e1fa0509e", "content-length": "3141", "date": "Wed, 30 Apr 2025 06:43:13 GMT"} })
122252025-04-30T06:43:13.175ZDEBGtest_snapshot_basic (BackgroundTasks): begin collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
122262025-04-30T06:43:13.175ZDEBGtest_snapshot_basic (BackgroundTasks): end collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
122272025-04-30T06:43:13.175ZDEBGtest_snapshot_basic (BackgroundTasks): finished collection background_task = inventory_collection
122282025-04-30T06:43:13.178ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
122292025-04-30T06:43:13.178ZTRACtest_snapshot_basic (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
122302025-04-30T06:43:13.178ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
122312025-04-30T06:43:13.178ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
122322025-04-30T06:43:13.179ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
122332025-04-30T06:43:13.181ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
122342025-04-30T06:43:13.186ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
122352025-04-30T06:43:13.186ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
122362025-04-30T06:43:13.186ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
122372025-04-30T06:43:13.187ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
122382025-04-30T06:43:13.219ZDEBGtest_snapshot_basic: attempt to fetch latest inventory collection result = Ok("found")
122392025-04-30T06:43:13.222ZWARNtest_snapshot_basic (datastore): Non-retryable transaction failure err = DatabaseError(UniqueViolation, "duplicate key value violates unique constraint \\"inv_caboose_pkey\\"") transaction = inventory_insert_collection
122402025-04-30T06:43:13.223ZWARNtest_snapshot_basic (BackgroundTasks): inventory collection failed background_task = inventory_collection error = failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint "inv_caboose_pkey"
122412025-04-30T06:43:13.223ZDEBGtest_snapshot_basic (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 694.468383ms iteration = 43 status = {"error":"failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint \\"inv_caboose_pkey\\""}
122422025-04-30T06:43:13.223ZDEBGtest_snapshot_basic (BackgroundTasks): activating background_task = inventory_collection iteration = 44 reason = Signaled
122432025-04-30T06:43:13.223ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
122442025-04-30T06:43:13.223ZTRACtest_snapshot_basic (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
122452025-04-30T06:43:13.223ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
122462025-04-30T06:43:13.223ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
122472025-04-30T06:43:13.225ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
122482025-04-30T06:43:13.240ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
122492025-04-30T06:43:13.246ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
122502025-04-30T06:43:13.248ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
122512025-04-30T06:43:13.248ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
122522025-04-30T06:43:13.249ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
122532025-04-30T06:43:13.253ZDEBGtest_snapshot_basic (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = [(d1ad44e5-18dd-4193-a70e-bf9042f0e0e1, 0)]
122542025-04-30T06:43:13.253ZTRACtest_snapshot_basic (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
122552025-04-30T06:43:13.254ZTRACtest_snapshot_basic (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: 91 ad 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 ............._mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:58902 req_id = 81485cce-b5e0-4123-b457-bdb82648a609
122562025-04-30T06:43:13.254ZDEBGtest_snapshot_basic (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 37293,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:58902 req_id = 81485cce-b5e0-4123-b457-bdb82648a609
122572025-04-30T06:43:13.254ZDEBGtest_snapshot_basic (store): query key key = _mgs._tcp
122582025-04-30T06:43:13.255ZDEBGtest_snapshot_basic (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
122592025-04-30T06:43:13.255ZDEBGtest_snapshot_basic (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:58902 query = LowerQuery { name: LowerName(Name("_mgs._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 41502, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = 81485cce-b5e0-4123-b457-bdb82648a609
122602025-04-30T06:43:13.255ZDEBGtest_snapshot_basic (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 41502, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 4258, tv_nsec: 54601637 } })
122612025-04-30T06:43:13.255ZTRACtest_snapshot_basic (DnsResolver): lookup_service_targets: looking up SRV target
122622025-04-30T06:43:13.256ZTRACtest_snapshot_basic (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: 37 58 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e 7X..........-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:50728 req_id = a27b91fc-423e-4bbb-b89b-7b527aedae58
122632025-04-30T06:43:13.256ZDEBGtest_snapshot_basic (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 14168,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:50728 req_id = a27b91fc-423e-4bbb-b89b-7b527aedae58
122642025-04-30T06:43:13.257ZDEBGtest_snapshot_basic (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
122652025-04-30T06:43:13.257ZDEBGtest_snapshot_basic (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:50728 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = a27b91fc-423e-4bbb-b89b-7b527aedae58
122662025-04-30T06:43:13.326ZTRACtest_snapshot_basic (DnsResolver): lookup_all_socket_v6 srv dns_name = _clickhouse-admin-keeper._tcp.control-plane.oxide.internal
122672025-04-30T06:43:13.328ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
122682025-04-30T06:43:13.328ZTRACtest_snapshot_basic (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
122692025-04-30T06:43:13.329ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
122702025-04-30T06:43:13.329ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {} }
122712025-04-30T06:43:13.329ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
122722025-04-30T06:43:13.330ZTRACtest_snapshot_basic (dns): buffer buffer = Length: 87 (0x57) bytes\n0000: 81 2a 01 00 00 01 00 00 00 00 00 01 18 5f 63 6c .*..........._cl\n0010: 69 63 6b 68 6f 75 73 65 2d 61 64 6d 69 6e 2d 6b ickhouse-admin-k\n0020: 65 65 70 65 72 04 5f 74 63 70 0d 63 6f 6e 74 72 eeper._tcp.contr\n0030: 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 69 ol-plane.oxide.i\n0040: 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 04 nternal..!....).\n0050: d0 00 00 00 00 00 00 ....... kind = dns peer_addr = [::1]:56482 req_id = 166942b0-9a11-4d8a-8cd4-1192e6125166
122732025-04-30T06:43:13.330ZDEBGtest_snapshot_basic (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 33066,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 24,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 97,\n 100,\n 109,\n 105,\n 110,\n 45,\n 107,\n 101,\n 101,\n 112,\n 101,\n 114,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:56482 req_id = 166942b0-9a11-4d8a-8cd4-1192e6125166
122742025-04-30T06:43:13.330ZDEBGtest_snapshot_basic (store): query key key = _clickhouse-admin-keeper._tcp
122752025-04-30T06:43:13.330ZERROtest_snapshot_basic (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 33066, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } NXDOMAIN: no records found for name: "_clickhouse-admin-keeper._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:56482 req_id = 166942b0-9a11-4d8a-8cd4-1192e6125166
122762025-04-30T06:43:13.330ZDEBGtest_snapshot_basic (BackgroundTasks): begin collection background_task = inventory_collection
122772025-04-30T06:43:13.330ZDEBGtest_snapshot_basic (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:41502
122782025-04-30T06:43:13.330ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/ignition
122792025-04-30T06:43:13.333ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
122802025-04-30T06:43:13.341ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
122812025-04-30T06:43:13.341ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
122822025-04-30T06:43:13.341ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {} }
122832025-04-30T06:43:13.342ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
122842025-04-30T06:43:13.342ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "1e4323fb-947c-48dc-ad52-a59a54df520b", "content-length": "835", "date": "Wed, 30 Apr 2025 06:43:13 GMT"} })
122852025-04-30T06:43:13.342ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/0
122862025-04-30T06:43:13.347ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
122872025-04-30T06:43:13.347ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {} }
122882025-04-30T06:43:13.348ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
122892025-04-30T06:43:13.348ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "1627c63f-d537-41df-a157-37d97f5eceba", "content-length": "734", "date": "Wed, 30 Apr 2025 06:43:13 GMT"} })
122902025-04-30T06:43:13.348ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/0/component/sp/caboose?firmware_slot=0
122912025-04-30T06:43:13.354ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "e91288fe-f0b4-4867-a4a0-f9819bae3ccd", "content-length": "111", "date": "Wed, 30 Apr 2025 06:43:13 GMT"} })
122922025-04-30T06:43:13.354ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/0/component/sp/caboose?firmware_slot=1
122932025-04-30T06:43:13.357ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "9d23fa8d-ae91-4edc-9eaa-800983592e7c", "content-length": "111", "date": "Wed, 30 Apr 2025 06:43:13 GMT"} })
122942025-04-30T06:43:13.357ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/0/component/rot/caboose?firmware_slot=0
122952025-04-30T06:43:13.359ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "b9bc2189-364e-40b5-a4ce-9642635be559", "content-length": "170", "date": "Wed, 30 Apr 2025 06:43:13 GMT"} })
122962025-04-30T06:43:13.359ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/0/component/rot/caboose?firmware_slot=1
122972025-04-30T06:43:13.361ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "6ccd964b-4bfc-4dd7-beda-0bddee8e033e", "content-length": "170", "date": "Wed, 30 Apr 2025 06:43:13 GMT"} })
122982025-04-30T06:43:13.361ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/0/component/stage0/caboose?firmware_slot=0
122992025-04-30T06:43:13.373ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "84d76265-9f43-422b-bea2-6f8857860749", "content-length": "179", "date": "Wed, 30 Apr 2025 06:43:13 GMT"} })
123002025-04-30T06:43:13.373ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/0/component/stage0/caboose?firmware_slot=1
123012025-04-30T06:43:13.376ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "61fb339d-515e-454b-820b-3c9cc417897e", "content-length": "179", "date": "Wed, 30 Apr 2025 06:43:13 GMT"} })
123022025-04-30T06:43:13.376ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/0/component/rot/cmpa
123032025-04-30T06:43:13.379ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "195c5050-bf93-4859-8e8a-bdd411f43742", "content-length": "702", "date": "Wed, 30 Apr 2025 06:43:13 GMT"} })
123042025-04-30T06:43:13.379ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/0/component/rot/cfpa
123052025-04-30T06:43:13.381ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "ebbbeddd-269f-46ce-b785-e421a99a4388", "content-length": "727", "date": "Wed, 30 Apr 2025 06:43:13 GMT"} })
123062025-04-30T06:43:13.381ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/0/component/rot/cfpa
123072025-04-30T06:43:13.383ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "581375eb-201d-46c1-954e-41b220ee6ee5", "content-length": "729", "date": "Wed, 30 Apr 2025 06:43:13 GMT"} })
123082025-04-30T06:43:13.383ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/0/component/rot/cfpa
123092025-04-30T06:43:13.398ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "07574b81-31bf-4121-8704-dccc947369b0", "content-length": "728", "date": "Wed, 30 Apr 2025 06:43:13 GMT"} })
123102025-04-30T06:43:13.398ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/1
123112025-04-30T06:43:13.399ZDEBGtest_snapshot_basic: attempt to fetch latest inventory collection result = Ok("found")
123122025-04-30T06:43:13.401ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "8ece7ada-d965-4751-a2cd-da0e2b1642ba", "content-length": "734", "date": "Wed, 30 Apr 2025 06:43:13 GMT"} })
123132025-04-30T06:43:13.401ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/1/component/sp/caboose?firmware_slot=0
123142025-04-30T06:43:13.404ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "fe4a4b09-bce7-41d1-a5ce-142bc9b62a97", "content-length": "111", "date": "Wed, 30 Apr 2025 06:43:13 GMT"} })
123152025-04-30T06:43:13.405ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/1/component/sp/caboose?firmware_slot=1
123162025-04-30T06:43:13.408ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "91693486-75ba-429c-8bd1-55e82f64d5e6", "content-length": "111", "date": "Wed, 30 Apr 2025 06:43:13 GMT"} })
123172025-04-30T06:43:13.408ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/1/component/rot/caboose?firmware_slot=0
123182025-04-30T06:43:13.410ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "fc8b4eb9-4cb2-4893-849d-cf6bde9eb7cd", "content-length": "170", "date": "Wed, 30 Apr 2025 06:43:13 GMT"} })
123192025-04-30T06:43:13.410ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/1/component/rot/caboose?firmware_slot=1
123202025-04-30T06:43:13.414ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "9f41dd80-333c-41c6-a66b-9aa2351f2f91", "content-length": "170", "date": "Wed, 30 Apr 2025 06:43:13 GMT"} })
123212025-04-30T06:43:13.414ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/1/component/stage0/caboose?firmware_slot=0
123222025-04-30T06:43:13.417ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "7435d174-6152-42b1-bf41-85012d8f1bb3", "content-length": "179", "date": "Wed, 30 Apr 2025 06:43:13 GMT"} })
123232025-04-30T06:43:13.417ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/1/component/stage0/caboose?firmware_slot=1
123242025-04-30T06:43:13.419ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "b9f853f2-7e1b-4173-8c99-c3e97372d1a1", "content-length": "179", "date": "Wed, 30 Apr 2025 06:43:13 GMT"} })
123252025-04-30T06:43:13.419ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/1/component/rot/cmpa
123262025-04-30T06:43:13.422ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "297a4098-8f97-4be3-b3fa-611bb9018520", "content-length": "702", "date": "Wed, 30 Apr 2025 06:43:13 GMT"} })
123272025-04-30T06:43:13.422ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/1/component/rot/cfpa
123282025-04-30T06:43:13.422ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "400a8902-a076-464a-9a3c-8514092811cb", "content-length": "727", "date": "Wed, 30 Apr 2025 06:43:13 GMT"} })
123292025-04-30T06:43:13.423ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/1/component/rot/cfpa
123302025-04-30T06:43:13.424ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "9bc65ae7-f348-4394-83c3-ad5899c4e892", "content-length": "729", "date": "Wed, 30 Apr 2025 06:43:13 GMT"} })
123312025-04-30T06:43:13.424ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/1/component/rot/cfpa
123322025-04-30T06:43:13.425ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "67d8dde9-b9f7-48ca-9dd1-7eb081a83e23", "content-length": "728", "date": "Wed, 30 Apr 2025 06:43:13 GMT"} })
123332025-04-30T06:43:13.425ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/0
123342025-04-30T06:43:13.427ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "882e6cbc-a529-47dc-be9d-50f9f9e2c43b", "content-length": "723", "date": "Wed, 30 Apr 2025 06:43:13 GMT"} })
123352025-04-30T06:43:13.427ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/0/component/sp/caboose?firmware_slot=0
123362025-04-30T06:43:13.429ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "ab720e23-458a-472c-b74e-47a84342df77", "content-length": "109", "date": "Wed, 30 Apr 2025 06:43:13 GMT"} })
123372025-04-30T06:43:13.430ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/0/component/sp/caboose?firmware_slot=1
123382025-04-30T06:43:13.432ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "9f75145e-1634-4567-b3b4-34b4786d4c0f", "content-length": "109", "date": "Wed, 30 Apr 2025 06:43:13 GMT"} })
123392025-04-30T06:43:13.432ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/0/component/rot/caboose?firmware_slot=0
123402025-04-30T06:43:13.434ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "688cdd41-c478-4850-a12f-c471b98825ee", "content-length": "169", "date": "Wed, 30 Apr 2025 06:43:13 GMT"} })
123412025-04-30T06:43:13.435ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/0/component/rot/caboose?firmware_slot=1
123422025-04-30T06:43:13.440ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "25eb1bd8-489f-4d05-b015-c69e95ed44a4", "content-length": "169", "date": "Wed, 30 Apr 2025 06:43:13 GMT"} })
123432025-04-30T06:43:13.440ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/0/component/stage0/caboose?firmware_slot=0
123442025-04-30T06:43:13.444ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "157d070c-824e-4482-87a6-bf045a029a70", "content-length": "178", "date": "Wed, 30 Apr 2025 06:43:13 GMT"} })
123452025-04-30T06:43:13.444ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/0/component/stage0/caboose?firmware_slot=1
123462025-04-30T06:43:13.448ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "48fbee67-8e24-4327-8f0e-d5d53882ded3", "content-length": "178", "date": "Wed, 30 Apr 2025 06:43:13 GMT"} })
123472025-04-30T06:43:13.448ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/0/component/rot/cmpa
123482025-04-30T06:43:13.451ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
123492025-04-30T06:43:13.451ZTRACtest_snapshot_basic (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
123502025-04-30T06:43:13.451ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
123512025-04-30T06:43:13.451ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {} }
123522025-04-30T06:43:13.452ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
123532025-04-30T06:43:13.453ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "90b59d6a-653e-482f-abed-926fd5d6be42", "content-length": "702", "date": "Wed, 30 Apr 2025 06:43:13 GMT"} })
123542025-04-30T06:43:13.453ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/0/component/rot/cfpa
123552025-04-30T06:43:13.455ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
123562025-04-30T06:43:13.464ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
123572025-04-30T06:43:13.464ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
123582025-04-30T06:43:13.464ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {} }
123592025-04-30T06:43:13.472ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
123602025-04-30T06:43:13.474ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "c70388ae-cd98-4127-bf2e-515a9a1a2800", "content-length": "727", "date": "Wed, 30 Apr 2025 06:43:13 GMT"} })
123612025-04-30T06:43:13.474ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/0/component/rot/cfpa
123622025-04-30T06:43:13.476ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
123632025-04-30T06:43:13.476ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {} }
123642025-04-30T06:43:13.477ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
123652025-04-30T06:43:13.478ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "74bd9e0c-4357-48af-9699-e2c7755dc8bb", "content-length": "729", "date": "Wed, 30 Apr 2025 06:43:13 GMT"} })
123662025-04-30T06:43:13.478ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/0/component/rot/cfpa
123672025-04-30T06:43:13.479ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "1138f6b1-d5de-4f4f-b963-f445060dd826", "content-length": "728", "date": "Wed, 30 Apr 2025 06:43:13 GMT"} })
123682025-04-30T06:43:13.479ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/1
123692025-04-30T06:43:13.481ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "8e6e85e9-bc0f-4689-8927-9aed8cd99bc4", "content-length": "723", "date": "Wed, 30 Apr 2025 06:43:13 GMT"} })
123702025-04-30T06:43:13.481ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/1/component/sp/caboose?firmware_slot=0
123712025-04-30T06:43:13.486ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "aebe88c7-a426-4d43-906b-fb5383e919f2", "content-length": "109", "date": "Wed, 30 Apr 2025 06:43:13 GMT"} })
123722025-04-30T06:43:13.486ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/1/component/sp/caboose?firmware_slot=1
123732025-04-30T06:43:13.494ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "bada807e-91aa-458b-9244-3238d718550c", "content-length": "109", "date": "Wed, 30 Apr 2025 06:43:13 GMT"} })
123742025-04-30T06:43:13.494ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/1/component/rot/caboose?firmware_slot=0
123752025-04-30T06:43:13.502ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "44402262-3993-4a13-b3f7-2e752cfca08d", "content-length": "169", "date": "Wed, 30 Apr 2025 06:43:13 GMT"} })
123762025-04-30T06:43:13.503ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/1/component/rot/caboose?firmware_slot=1
123772025-04-30T06:43:13.522ZDEBGtest_snapshot_basic: attempt to fetch latest inventory collection result = Ok("found")
123782025-04-30T06:43:13.542ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "a24ffe80-89af-4881-a262-4c2ea184c778", "content-length": "169", "date": "Wed, 30 Apr 2025 06:43:13 GMT"} })
123792025-04-30T06:43:13.542ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/1/component/stage0/caboose?firmware_slot=0
123802025-04-30T06:43:13.572ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "164c228e-f2bc-444f-b5c3-ed37f370cfe3", "content-length": "178", "date": "Wed, 30 Apr 2025 06:43:13 GMT"} })
123812025-04-30T06:43:13.572ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/1/component/stage0/caboose?firmware_slot=1
123822025-04-30T06:43:13.581ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
123832025-04-30T06:43:13.581ZTRACtest_snapshot_basic (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
123842025-04-30T06:43:13.581ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
123852025-04-30T06:43:13.581ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {} }
123862025-04-30T06:43:13.582ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
123872025-04-30T06:43:13.585ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
123882025-04-30T06:43:13.596ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
123892025-04-30T06:43:13.596ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
123902025-04-30T06:43:13.596ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {} }
123912025-04-30T06:43:13.596ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
123922025-04-30T06:43:13.600ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
123932025-04-30T06:43:13.600ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {} }
123942025-04-30T06:43:13.601ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
123952025-04-30T06:43:13.601ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "7142bb89-e7ba-4701-84ed-f72fbc348e62", "content-length": "178", "date": "Wed, 30 Apr 2025 06:43:13 GMT"} })
123962025-04-30T06:43:13.601ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/1/component/rot/cmpa
123972025-04-30T06:43:13.603ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "3cabe810-625c-46c1-ae68-5107fdacff05", "content-length": "702", "date": "Wed, 30 Apr 2025 06:43:13 GMT"} })
123982025-04-30T06:43:13.603ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/1/component/rot/cfpa
123992025-04-30T06:43:13.607ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "8d061df7-05e3-4220-92d1-1595a36975cf", "content-length": "727", "date": "Wed, 30 Apr 2025 06:43:13 GMT"} })
124002025-04-30T06:43:13.607ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/1/component/rot/cfpa
124012025-04-30T06:43:13.609ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "85ddf6b5-e679-4f2c-9b67-ba6b34c4b49c", "content-length": "729", "date": "Wed, 30 Apr 2025 06:43:13 GMT"} })
124022025-04-30T06:43:13.609ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/1/component/rot/cfpa
124032025-04-30T06:43:13.611ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "c5ddc119-b756-462c-9aeb-206f59083993", "content-length": "728", "date": "Wed, 30 Apr 2025 06:43:13 GMT"} })
124042025-04-30T06:43:13.620ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
124052025-04-30T06:43:13.620ZTRACtest_snapshot_basic (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
124062025-04-30T06:43:13.620ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
124072025-04-30T06:43:13.620ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
124082025-04-30T06:43:13.621ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
124092025-04-30T06:43:13.623ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
124102025-04-30T06:43:13.628ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
124112025-04-30T06:43:13.628ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
124122025-04-30T06:43:13.628ZTRACtest_snapshot_basic (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
124132025-04-30T06:43:13.628ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
124142025-04-30T06:43:13.628ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
124152025-04-30T06:43:13.633ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
124162025-04-30T06:43:13.638ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
124172025-04-30T06:43:13.642ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
124182025-04-30T06:43:13.642ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
124192025-04-30T06:43:13.642ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
124202025-04-30T06:43:13.643ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
124212025-04-30T06:43:13.712ZDEBGtest_snapshot_basic (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:41923
124222025-04-30T06:43:13.712ZDEBGtest_snapshot_basic (BackgroundTasks): client request SledAgent = http://[::1]:41923 background_task = inventory_collection body = None method = GET uri = http://[::1]:41923/inventory
124232025-04-30T06:43:13.714ZINFOtest_snapshot_basic (dropshot): accepted connection local_addr = [::1]:41923 remote_addr = [::1]:52817 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
124242025-04-30T06:43:13.716ZTRACtest_snapshot_basic (dropshot): incoming request local_addr = [::1]:41923 method = GET remote_addr = [::1]:52817 req_id = fdd13ce9-97f0-48ef-a638-5c56b4c440bf sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
124252025-04-30T06:43:13.717ZINFOtest_snapshot_basic (dropshot): request completed latency_us = 564 local_addr = [::1]:41923 method = GET remote_addr = [::1]:52817 req_id = fdd13ce9-97f0-48ef-a638-5c56b4c440bf response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
124262025-04-30T06:43:13.718ZDEBGtest_snapshot_basic (BackgroundTasks): client response SledAgent = http://[::1]:41923 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:41923/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "fdd13ce9-97f0-48ef-a638-5c56b4c440bf", "content-length": "3141", "date": "Wed, 30 Apr 2025 06:43:13 GMT"} })
124272025-04-30T06:43:13.722ZDEBGtest_snapshot_basic (BackgroundTasks): begin collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
124282025-04-30T06:43:13.722ZDEBGtest_snapshot_basic (BackgroundTasks): end collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
124292025-04-30T06:43:13.722ZDEBGtest_snapshot_basic (BackgroundTasks): finished collection background_task = inventory_collection
124302025-04-30T06:43:13.725ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
124312025-04-30T06:43:13.725ZTRACtest_snapshot_basic (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
124322025-04-30T06:43:13.725ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
124332025-04-30T06:43:13.725ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
124342025-04-30T06:43:13.726ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
124352025-04-30T06:43:13.727ZINFOtest_snapshot_basic (oximeter-agent): refreshing list of producers from Nexus collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 file = oximeter/collector/src/agent.rs:562
124362025-04-30T06:43:13.727ZDEBGtest_snapshot_basic (oximeter): client request body = None method = GET uri = http://127.0.0.1:61468/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
124372025-04-30T06:43:13.729ZTRACtest_snapshot_basic (dropshot_internal): incoming request local_addr = 127.0.0.1:61468 method = GET remote_addr = 127.0.0.1:64332 req_id = 846ea938-16f4-4a66-832d-0c45255fd775 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
124382025-04-30T06:43:13.729ZTRACtest_snapshot_basic (dropshot_internal): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:61468 method = GET remote_addr = 127.0.0.1:64332 req_id = 846ea938-16f4-4a66-832d-0c45255fd775 resource = Database uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
124392025-04-30T06:43:13.730ZDEBGtest_snapshot_basic (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:61468 method = GET remote_addr = 127.0.0.1:64332 req_id = 846ea938-16f4-4a66-832d-0c45255fd775 roles = RoleSet { roles: {} } uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
124402025-04-30T06:43:13.731ZDEBGtest_snapshot_basic (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:61468 method = GET remote_addr = 127.0.0.1:64332 req_id = 846ea938-16f4-4a66-832d-0c45255fd775 resource = Database result = Ok(()) uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
124412025-04-30T06:43:13.731ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
124422025-04-30T06:43:13.737ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
124432025-04-30T06:43:13.737ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
124442025-04-30T06:43:13.737ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
124452025-04-30T06:43:13.738ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
124462025-04-30T06:43:13.744ZINFOtest_snapshot_basic (dropshot_internal): request completed latency_us = 14592 local_addr = 127.0.0.1:61468 method = GET remote_addr = 127.0.0.1:64332 req_id = 846ea938-16f4-4a66-832d-0c45255fd775 response_code = 200 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
124472025-04-30T06:43:13.744ZDEBGtest_snapshot_basic (oximeter): client response result = Ok(Response { url: "http://127.0.0.1:61468/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending", status: 200, headers: {"content-type": "application/json", "x-request-id": "846ea938-16f4-4a66-832d-0c45255fd775", "content-length": "405", "date": "Wed, 30 Apr 2025 06:43:13 GMT"} })
124482025-04-30T06:43:13.744ZDEBGtest_snapshot_basic (oximeter): client request body = None method = GET uri = http://127.0.0.1:61468/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
124492025-04-30T06:43:13.745ZDEBGtest_snapshot_basic: attempt to fetch latest inventory collection result = Ok("found")
124502025-04-30T06:43:13.745ZTRACtest_snapshot_basic (dropshot_internal): incoming request local_addr = 127.0.0.1:61468 method = GET remote_addr = 127.0.0.1:64332 req_id = 021a71c8-32df-4af1-bbeb-02c4cbbe08d6 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
124512025-04-30T06:43:13.745ZTRACtest_snapshot_basic (dropshot_internal): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:61468 method = GET remote_addr = 127.0.0.1:64332 req_id = 021a71c8-32df-4af1-bbeb-02c4cbbe08d6 resource = Database uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
124522025-04-30T06:43:13.745ZDEBGtest_snapshot_basic (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:61468 method = GET remote_addr = 127.0.0.1:64332 req_id = 021a71c8-32df-4af1-bbeb-02c4cbbe08d6 roles = RoleSet { roles: {} } uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
124532025-04-30T06:43:13.747ZDEBGtest_snapshot_basic (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:61468 method = GET remote_addr = 127.0.0.1:64332 req_id = 021a71c8-32df-4af1-bbeb-02c4cbbe08d6 resource = Database result = Ok(()) uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
124542025-04-30T06:43:13.755ZINFOtest_snapshot_basic (dropshot_internal): request completed latency_us = 9695 local_addr = 127.0.0.1:61468 method = GET remote_addr = 127.0.0.1:64332 req_id = 021a71c8-32df-4af1-bbeb-02c4cbbe08d6 response_code = 200 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
124552025-04-30T06:43:13.755ZDEBGtest_snapshot_basic (oximeter): client response result = Ok(Response { url: "http://127.0.0.1:61468/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D", status: 200, headers: {"content-type": "application/json", "x-request-id": "021a71c8-32df-4af1-bbeb-02c4cbbe08d6", "content-length": "29", "date": "Wed, 30 Apr 2025 06:43:13 GMT"} })
124562025-04-30T06:43:13.756ZTRACtest_snapshot_basic (oximeter-agent): ignoring request to update existing metric producer, since the endpoint information is the same as the existing collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = 354a52d4-3078-4706-81ce-12c32f6bbeaa
124572025-04-30T06:43:13.756ZTRACtest_snapshot_basic (oximeter-agent): ignoring request to update existing metric producer, since the endpoint information is the same as the existing collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
124582025-04-30T06:43:13.756ZINFOtest_snapshot_basic (oximeter-agent): refreshed list of producers from Nexus collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 file = oximeter/collector/src/agent.rs:637 n_current_tasks = 2 n_pruned_tasks = 0
124592025-04-30T06:43:13.805ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
124602025-04-30T06:43:13.805ZTRACtest_snapshot_basic (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
124612025-04-30T06:43:13.805ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
124622025-04-30T06:43:13.805ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {} }
124632025-04-30T06:43:13.806ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
124642025-04-30T06:43:13.806ZDEBGtest_snapshot_basic (collection-task): sent timer-based collection request to the collection task collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
124652025-04-30T06:43:13.806ZDEBGtest_snapshot_basic (collection-task): collecting from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
124662025-04-30T06:43:13.808ZTRACtest_snapshot_basic (collection-task): sent collection request to producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
124672025-04-30T06:43:13.808ZDEBGtest_snapshot_basic (collection-task): collected results from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 n_results = 1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
124682025-04-30T06:43:13.809ZTRACtest_snapshot_basic (collection-task): forwarded results to main collection loop collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
124692025-04-30T06:43:13.809ZTRACtest_snapshot_basic (collection-task): top of inner collection loop, waiting for next request collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
124702025-04-30T06:43:13.809ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
124712025-04-30T06:43:13.814ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
124722025-04-30T06:43:13.814ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
124732025-04-30T06:43:13.814ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {} }
124742025-04-30T06:43:13.815ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
124752025-04-30T06:43:13.862ZWARNtest_snapshot_basic (datastore): Non-retryable transaction failure err = DatabaseError(UniqueViolation, "duplicate key value violates unique constraint \\"inv_caboose_pkey\\"") transaction = inventory_insert_collection
124762025-04-30T06:43:13.863ZWARNtest_snapshot_basic (BackgroundTasks): inventory collection failed background_task = inventory_collection error = failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint "inv_caboose_pkey"
124772025-04-30T06:43:13.863ZDEBGtest_snapshot_basic (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 640.099878ms iteration = 44 status = {"error":"failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint \\"inv_caboose_pkey\\""}
124782025-04-30T06:43:13.863ZDEBGtest_snapshot_basic (BackgroundTasks): activating background_task = inventory_collection iteration = 45 reason = Signaled
124792025-04-30T06:43:13.863ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
124802025-04-30T06:43:13.863ZTRACtest_snapshot_basic (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
124812025-04-30T06:43:13.863ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
124822025-04-30T06:43:13.863ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
124832025-04-30T06:43:13.867ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
124842025-04-30T06:43:13.869ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
124852025-04-30T06:43:13.869ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {} }
124862025-04-30T06:43:13.871ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
124872025-04-30T06:43:13.873ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
124882025-04-30T06:43:13.880ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
124892025-04-30T06:43:13.880ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
124902025-04-30T06:43:13.880ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
124912025-04-30T06:43:13.881ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
124922025-04-30T06:43:13.884ZDEBGtest_snapshot_basic (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = [(d1ad44e5-18dd-4193-a70e-bf9042f0e0e1, 0)]
124932025-04-30T06:43:13.885ZTRACtest_snapshot_basic (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
124942025-04-30T06:43:13.887ZTRACtest_snapshot_basic (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: a8 fd 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 ............._mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:51405 req_id = 194e947a-45d9-4fc6-a49d-67cbc7d25943
124952025-04-30T06:43:13.887ZDEBGtest_snapshot_basic (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 43261,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:51405 req_id = 194e947a-45d9-4fc6-a49d-67cbc7d25943
124962025-04-30T06:43:13.887ZDEBGtest_snapshot_basic (store): query key key = _mgs._tcp
124972025-04-30T06:43:13.887ZDEBGtest_snapshot_basic (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
124982025-04-30T06:43:13.888ZDEBGtest_snapshot_basic (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:51405 query = LowerQuery { name: LowerName(Name("_mgs._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 41502, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = 194e947a-45d9-4fc6-a49d-67cbc7d25943
124992025-04-30T06:43:13.889ZDEBGtest_snapshot_basic (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 41502, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 4258, tv_nsec: 688446833 } })
125002025-04-30T06:43:13.889ZTRACtest_snapshot_basic (DnsResolver): lookup_service_targets: looking up SRV target
125012025-04-30T06:43:13.890ZTRACtest_snapshot_basic (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: 7f 2d 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e .-..........-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:55377 req_id = 1d6c38c9-c668-441d-8a7d-f3106c23f32f
125022025-04-30T06:43:13.895ZDEBGtest_snapshot_basic (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 32557,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:55377 req_id = 1d6c38c9-c668-441d-8a7d-f3106c23f32f
125032025-04-30T06:43:13.897ZDEBGtest_snapshot_basic (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
125042025-04-30T06:43:13.897ZDEBGtest_snapshot_basic (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:55377 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = 1d6c38c9-c668-441d-8a7d-f3106c23f32f
125052025-04-30T06:43:13.972ZTRACtest_snapshot_basic (DnsResolver): lookup_all_socket_v6 srv dns_name = _clickhouse-admin-keeper._tcp.control-plane.oxide.internal
125062025-04-30T06:43:13.972ZTRACtest_snapshot_basic (dns): buffer buffer = Length: 87 (0x57) bytes\n0000: 2f dc 01 00 00 01 00 00 00 00 00 01 18 5f 63 6c /............_cl\n0010: 69 63 6b 68 6f 75 73 65 2d 61 64 6d 69 6e 2d 6b ickhouse-admin-k\n0020: 65 65 70 65 72 04 5f 74 63 70 0d 63 6f 6e 74 72 eeper._tcp.contr\n0030: 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 69 ol-plane.oxide.i\n0040: 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 04 nternal..!....).\n0050: d0 00 00 00 00 00 00 ....... kind = dns peer_addr = [::1]:58160 req_id = 59809633-e090-470b-a8b3-593b2fbaa1bd
125072025-04-30T06:43:13.973ZDEBGtest_snapshot_basic (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 12252,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 24,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 97,\n 100,\n 109,\n 105,\n 110,\n 45,\n 107,\n 101,\n 101,\n 112,\n 101,\n 114,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:58160 req_id = 59809633-e090-470b-a8b3-593b2fbaa1bd
125082025-04-30T06:43:13.973ZDEBGtest_snapshot_basic (store): query key key = _clickhouse-admin-keeper._tcp
125092025-04-30T06:43:13.973ZERROtest_snapshot_basic (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 12252, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } NXDOMAIN: no records found for name: "_clickhouse-admin-keeper._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:58160 req_id = 59809633-e090-470b-a8b3-593b2fbaa1bd
125102025-04-30T06:43:13.973ZDEBGtest_snapshot_basic (BackgroundTasks): begin collection background_task = inventory_collection
125112025-04-30T06:43:13.973ZDEBGtest_snapshot_basic (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:41502
125122025-04-30T06:43:13.973ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/ignition
125132025-04-30T06:43:13.976ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "c72e4322-5cc0-4392-97f6-c9ae22b6bb1a", "content-length": "835", "date": "Wed, 30 Apr 2025 06:43:13 GMT"} })
125142025-04-30T06:43:13.976ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/0
125152025-04-30T06:43:13.979ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "a162ea0f-cdaf-476e-84c1-c43f44a133c8", "content-length": "734", "date": "Wed, 30 Apr 2025 06:43:13 GMT"} })
125162025-04-30T06:43:13.979ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/0/component/sp/caboose?firmware_slot=0
125172025-04-30T06:43:13.981ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "b267839c-19f9-4a2d-8554-6bae12390d42", "content-length": "111", "date": "Wed, 30 Apr 2025 06:43:13 GMT"} })
125182025-04-30T06:43:13.982ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/0/component/sp/caboose?firmware_slot=1
125192025-04-30T06:43:13.989ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "b5ccfcf2-51e5-4984-99f1-8f6a72f36704", "content-length": "111", "date": "Wed, 30 Apr 2025 06:43:13 GMT"} })
125202025-04-30T06:43:13.989ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/0/component/rot/caboose?firmware_slot=0
125212025-04-30T06:43:13.990ZDEBGtest_snapshot_basic: attempt to fetch latest inventory collection result = Ok("found")
125222025-04-30T06:43:14.004ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "3601e28e-989e-4235-886f-a9266d844cba", "content-length": "170", "date": "Wed, 30 Apr 2025 06:43:13 GMT"} })
125232025-04-30T06:43:14.005ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/0/component/rot/caboose?firmware_slot=1
125242025-04-30T06:43:14.007ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "d6574879-a50d-4448-83e1-249ef269380a", "content-length": "170", "date": "Wed, 30 Apr 2025 06:43:13 GMT"} })
125252025-04-30T06:43:14.007ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/0/component/stage0/caboose?firmware_slot=0
125262025-04-30T06:43:14.009ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "3be40561-d2e9-4aae-98eb-3b47ca7f0a1b", "content-length": "179", "date": "Wed, 30 Apr 2025 06:43:13 GMT"} })
125272025-04-30T06:43:14.009ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/0/component/stage0/caboose?firmware_slot=1
125282025-04-30T06:43:14.011ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "aa746acc-6a2c-48f0-9db3-ce16ff46c5d6", "content-length": "179", "date": "Wed, 30 Apr 2025 06:43:13 GMT"} })
125292025-04-30T06:43:14.011ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/0/component/rot/cmpa
125302025-04-30T06:43:14.013ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "ba69d13c-c93b-441c-b859-2c5ff29c6612", "content-length": "702", "date": "Wed, 30 Apr 2025 06:43:13 GMT"} })
125312025-04-30T06:43:14.013ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/0/component/rot/cfpa
125322025-04-30T06:43:14.014ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "e4263a0c-565a-41ee-8423-db7fb5e97af8", "content-length": "727", "date": "Wed, 30 Apr 2025 06:43:13 GMT"} })
125332025-04-30T06:43:14.015ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/0/component/rot/cfpa
125342025-04-30T06:43:14.016ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "b10c16cf-9643-48b8-abe8-8e89b3f07465", "content-length": "729", "date": "Wed, 30 Apr 2025 06:43:13 GMT"} })
125352025-04-30T06:43:14.016ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/0/component/rot/cfpa
125362025-04-30T06:43:14.017ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "6d14f40a-866c-4a38-9162-b3f8ab0d27af", "content-length": "728", "date": "Wed, 30 Apr 2025 06:43:13 GMT"} })
125372025-04-30T06:43:14.017ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/1
125382025-04-30T06:43:14.018ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "a2cedd79-187b-4f5e-b18e-b8f8983a0a56", "content-length": "734", "date": "Wed, 30 Apr 2025 06:43:13 GMT"} })
125392025-04-30T06:43:14.018ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/1/component/sp/caboose?firmware_slot=0
125402025-04-30T06:43:14.021ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "9f3d497e-7577-43d2-8c0c-91247c9e0837", "content-length": "111", "date": "Wed, 30 Apr 2025 06:43:13 GMT"} })
125412025-04-30T06:43:14.021ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/1/component/sp/caboose?firmware_slot=1
125422025-04-30T06:43:14.024ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "45818596-eab0-427d-92ea-0b8019ba1339", "content-length": "111", "date": "Wed, 30 Apr 2025 06:43:13 GMT"} })
125432025-04-30T06:43:14.024ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/1/component/rot/caboose?firmware_slot=0
125442025-04-30T06:43:14.026ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "b34c0e73-d5c7-418e-967a-5a2db613f683", "content-length": "170", "date": "Wed, 30 Apr 2025 06:43:13 GMT"} })
125452025-04-30T06:43:14.026ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/1/component/rot/caboose?firmware_slot=1
125462025-04-30T06:43:14.028ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "9a9c184f-f9b5-45e6-ac83-5b142ccbec11", "content-length": "170", "date": "Wed, 30 Apr 2025 06:43:13 GMT"} })
125472025-04-30T06:43:14.028ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/1/component/stage0/caboose?firmware_slot=0
125482025-04-30T06:43:14.030ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "20111b22-fe23-4b7e-95f2-9fa3a3fc167f", "content-length": "179", "date": "Wed, 30 Apr 2025 06:43:13 GMT"} })
125492025-04-30T06:43:14.030ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/1/component/stage0/caboose?firmware_slot=1
125502025-04-30T06:43:14.034ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "0ea05c22-0d7c-48ae-b271-a9a0a3e338d4", "content-length": "179", "date": "Wed, 30 Apr 2025 06:43:13 GMT"} })
125512025-04-30T06:43:14.034ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/1/component/rot/cmpa
125522025-04-30T06:43:14.035ZDEBGtest_snapshot_basic (results-sink): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
125532025-04-30T06:43:14.035ZDEBGtest_snapshot_basic (results-sink-cluster): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
125542025-04-30T06:43:14.036ZDEBGtest_snapshot_basic (clickhouse-client): unrolling 1 total samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 6ad06c6b-9a6e-4ed1-b3f4-5d2e9da3f8a7
125552025-04-30T06:43:14.037ZTRACtest_snapshot_basic (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 6ad06c6b-9a6e-4ed1-b3f4-5d2e9da3f8a7 n_columns = 4 n_rows = 2 sql = INSERT INTO oximeter.fields_string FORMAT Native
125562025-04-30T06:43:14.038ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "50ba8e5e-2ba2-431d-85e6-8f6d912e46e1", "content-length": "702", "date": "Wed, 30 Apr 2025 06:43:13 GMT"} })
125572025-04-30T06:43:14.038ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/1/component/rot/cfpa
125582025-04-30T06:43:14.040ZDEBGtest_snapshot_basic (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 6ad06c6b-9a6e-4ed1-b3f4-5d2e9da3f8a7 n_rows = 2 table_name = fields_string
125592025-04-30T06:43:14.040ZTRACtest_snapshot_basic (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 6ad06c6b-9a6e-4ed1-b3f4-5d2e9da3f8a7 n_columns = 4 n_rows = 1 sql = INSERT INTO oximeter.measurements_i64 FORMAT Native
125602025-04-30T06:43:14.040ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "e4071844-0c0c-41a3-a996-70a6fd1a4d10", "content-length": "727", "date": "Wed, 30 Apr 2025 06:43:13 GMT"} })
125612025-04-30T06:43:14.041ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/1/component/rot/cfpa
125622025-04-30T06:43:14.041ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
125632025-04-30T06:43:14.041ZTRACtest_snapshot_basic (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
125642025-04-30T06:43:14.041ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
125652025-04-30T06:43:14.041ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {} }
125662025-04-30T06:43:14.042ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
125672025-04-30T06:43:14.044ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "52ae0277-10ed-4620-9abf-02a666fa4104", "content-length": "729", "date": "Wed, 30 Apr 2025 06:43:13 GMT"} })
125682025-04-30T06:43:14.044ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/1/component/rot/cfpa
125692025-04-30T06:43:14.045ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
125702025-04-30T06:43:14.052ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
125712025-04-30T06:43:14.052ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
125722025-04-30T06:43:14.052ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {} }
125732025-04-30T06:43:14.052ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
125742025-04-30T06:43:14.052ZDEBGtest_snapshot_basic (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 6ad06c6b-9a6e-4ed1-b3f4-5d2e9da3f8a7 n_rows = 1 table_name = measurements_i64
125752025-04-30T06:43:14.053ZTRACtest_snapshot_basic (results-sink): successfully inserted samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
125762025-04-30T06:43:14.055ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "f2e4590f-5df3-46f1-af50-e0d0f8bb5c0e", "content-length": "728", "date": "Wed, 30 Apr 2025 06:43:13 GMT"} })
125772025-04-30T06:43:14.055ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/0
125782025-04-30T06:43:14.055ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
125792025-04-30T06:43:14.055ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {} }
125802025-04-30T06:43:14.056ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
125812025-04-30T06:43:14.059ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "78fd2a15-2d58-40f7-824d-41a3aa70dc14", "content-length": "723", "date": "Wed, 30 Apr 2025 06:43:13 GMT"} })
125822025-04-30T06:43:14.059ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/0/component/sp/caboose?firmware_slot=0
125832025-04-30T06:43:14.075ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "46597c70-0fa8-4c3f-a1be-99a7cf7753ee", "content-length": "109", "date": "Wed, 30 Apr 2025 06:43:13 GMT"} })
125842025-04-30T06:43:14.075ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/0/component/sp/caboose?firmware_slot=1
125852025-04-30T06:43:14.081ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "933804e0-2bde-44af-9ff6-63ad19837462", "content-length": "109", "date": "Wed, 30 Apr 2025 06:43:13 GMT"} })
125862025-04-30T06:43:14.081ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/0/component/rot/caboose?firmware_slot=0
125872025-04-30T06:43:14.085ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "a71fda69-a163-424d-945b-07fdab1aadf2", "content-length": "169", "date": "Wed, 30 Apr 2025 06:43:13 GMT"} })
125882025-04-30T06:43:14.085ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/0/component/rot/caboose?firmware_slot=1
125892025-04-30T06:43:14.087ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "efc0ae14-c76b-4eb6-beff-b5f23a46540d", "content-length": "169", "date": "Wed, 30 Apr 2025 06:43:13 GMT"} })
125902025-04-30T06:43:14.087ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/0/component/stage0/caboose?firmware_slot=0
125912025-04-30T06:43:14.090ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "516f5b48-d6a6-49c5-b55e-fcb501ec5ec0", "content-length": "178", "date": "Wed, 30 Apr 2025 06:43:13 GMT"} })
125922025-04-30T06:43:14.091ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/0/component/stage0/caboose?firmware_slot=1
125932025-04-30T06:43:14.094ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "e22251bd-e6a1-4817-bb5f-740d673fcbdd", "content-length": "178", "date": "Wed, 30 Apr 2025 06:43:13 GMT"} })
125942025-04-30T06:43:14.094ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/0/component/rot/cmpa
125952025-04-30T06:43:14.095ZDEBGtest_snapshot_basic: attempt to fetch latest inventory collection result = Ok("found")
125962025-04-30T06:43:14.096ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "b74a889f-0bed-4ba4-8030-d222ebf140ff", "content-length": "702", "date": "Wed, 30 Apr 2025 06:43:13 GMT"} })
125972025-04-30T06:43:14.096ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/0/component/rot/cfpa
125982025-04-30T06:43:14.098ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "104d3d4b-445e-47eb-8a0a-3513fa5f0e24", "content-length": "727", "date": "Wed, 30 Apr 2025 06:43:13 GMT"} })
125992025-04-30T06:43:14.098ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/0/component/rot/cfpa
126002025-04-30T06:43:14.100ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "0c3621c0-25cd-4dce-9c85-1b56c827a94c", "content-length": "729", "date": "Wed, 30 Apr 2025 06:43:13 GMT"} })
126012025-04-30T06:43:14.101ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/0/component/rot/cfpa
126022025-04-30T06:43:14.103ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "df488ed3-58f5-456c-8bd9-864a10dc682f", "content-length": "728", "date": "Wed, 30 Apr 2025 06:43:13 GMT"} })
126032025-04-30T06:43:14.103ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/1
126042025-04-30T06:43:14.104ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "e8681634-6d5b-43ed-91f7-b183f7f1d628", "content-length": "723", "date": "Wed, 30 Apr 2025 06:43:13 GMT"} })
126052025-04-30T06:43:14.105ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/1/component/sp/caboose?firmware_slot=0
126062025-04-30T06:43:14.107ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "ca5873c8-c733-4635-a08b-c255df22db67", "content-length": "109", "date": "Wed, 30 Apr 2025 06:43:13 GMT"} })
126072025-04-30T06:43:14.107ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/1/component/sp/caboose?firmware_slot=1
126082025-04-30T06:43:14.109ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "4c2bd327-c615-4dbd-a238-4db406c1141b", "content-length": "109", "date": "Wed, 30 Apr 2025 06:43:13 GMT"} })
126092025-04-30T06:43:14.109ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/1/component/rot/caboose?firmware_slot=0
126102025-04-30T06:43:14.111ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "ec991473-8249-4222-8900-eb8c5614a253", "content-length": "169", "date": "Wed, 30 Apr 2025 06:43:13 GMT"} })
126112025-04-30T06:43:14.111ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/1/component/rot/caboose?firmware_slot=1
126122025-04-30T06:43:14.114ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "de78376d-030d-48c7-a272-a03bfe15b7ef", "content-length": "169", "date": "Wed, 30 Apr 2025 06:43:13 GMT"} })
126132025-04-30T06:43:14.115ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/1/component/stage0/caboose?firmware_slot=0
126142025-04-30T06:43:14.119ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "207d3024-5684-41dc-84a4-39504f79a843", "content-length": "178", "date": "Wed, 30 Apr 2025 06:43:13 GMT"} })
126152025-04-30T06:43:14.119ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/1/component/stage0/caboose?firmware_slot=1
126162025-04-30T06:43:14.122ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "f2b8ccaa-49fe-4bee-b51f-d863ddae91a2", "content-length": "178", "date": "Wed, 30 Apr 2025 06:43:13 GMT"} })
126172025-04-30T06:43:14.122ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/1/component/rot/cmpa
126182025-04-30T06:43:14.123ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "0d689938-9951-44fb-afdc-36749eea50bc", "content-length": "702", "date": "Wed, 30 Apr 2025 06:43:13 GMT"} })
126192025-04-30T06:43:14.123ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/1/component/rot/cfpa
126202025-04-30T06:43:14.126ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "59f1195a-341f-4914-b77e-132d8c5987b5", "content-length": "727", "date": "Wed, 30 Apr 2025 06:43:13 GMT"} })
126212025-04-30T06:43:14.126ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/1/component/rot/cfpa
126222025-04-30T06:43:14.128ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "4de7c923-258b-4034-9a7b-7848c7dbae2e", "content-length": "729", "date": "Wed, 30 Apr 2025 06:43:13 GMT"} })
126232025-04-30T06:43:14.128ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/1/component/rot/cfpa
126242025-04-30T06:43:14.130ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "78e0c762-e2c9-4466-b14f-6ceaf67ac413", "content-length": "728", "date": "Wed, 30 Apr 2025 06:43:13 GMT"} })
126252025-04-30T06:43:14.130ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
126262025-04-30T06:43:14.130ZTRACtest_snapshot_basic (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
126272025-04-30T06:43:14.130ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
126282025-04-30T06:43:14.130ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
126292025-04-30T06:43:14.131ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
126302025-04-30T06:43:14.136ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
126312025-04-30T06:43:14.141ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
126322025-04-30T06:43:14.141ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
126332025-04-30T06:43:14.141ZTRACtest_snapshot_basic (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
126342025-04-30T06:43:14.141ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
126352025-04-30T06:43:14.141ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
126362025-04-30T06:43:14.142ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
126372025-04-30T06:43:14.143ZWARNtest_snapshot_basic (results-sink-cluster): failed to insert some results into metric DB collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 error = Failed to check out connection to database: No backends found for this service file = oximeter/collector/src/results_sink.rs:92
126382025-04-30T06:43:14.146ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
126392025-04-30T06:43:14.162ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
126402025-04-30T06:43:14.163ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
126412025-04-30T06:43:14.163ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
126422025-04-30T06:43:14.164ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
126432025-04-30T06:43:14.164ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
126442025-04-30T06:43:14.164ZTRACtest_snapshot_basic (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
126452025-04-30T06:43:14.165ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
126462025-04-30T06:43:14.165ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {} }
126472025-04-30T06:43:14.165ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
126482025-04-30T06:43:14.231ZDEBGtest_snapshot_basic (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:41923
126492025-04-30T06:43:14.231ZDEBGtest_snapshot_basic (BackgroundTasks): client request SledAgent = http://[::1]:41923 background_task = inventory_collection body = None method = GET uri = http://[::1]:41923/inventory
126502025-04-30T06:43:14.231ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
126512025-04-30T06:43:14.240ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
126522025-04-30T06:43:14.240ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
126532025-04-30T06:43:14.240ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {} }
126542025-04-30T06:43:14.241ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
126552025-04-30T06:43:14.242ZINFOtest_snapshot_basic (dropshot): accepted connection local_addr = [::1]:41923 remote_addr = [::1]:38049 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
126562025-04-30T06:43:14.245ZTRACtest_snapshot_basic (dropshot): incoming request local_addr = [::1]:41923 method = GET remote_addr = [::1]:38049 req_id = 4311668c-6b7a-43e1-b686-c6ab3d3cc8d1 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
126572025-04-30T06:43:14.246ZINFOtest_snapshot_basic (dropshot): request completed latency_us = 1239 local_addr = [::1]:41923 method = GET remote_addr = [::1]:38049 req_id = 4311668c-6b7a-43e1-b686-c6ab3d3cc8d1 response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
126582025-04-30T06:43:14.246ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
126592025-04-30T06:43:14.246ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {} }
126602025-04-30T06:43:14.247ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
126612025-04-30T06:43:14.247ZDEBGtest_snapshot_basic (BackgroundTasks): client response SledAgent = http://[::1]:41923 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:41923/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "4311668c-6b7a-43e1-b686-c6ab3d3cc8d1", "content-length": "3141", "date": "Wed, 30 Apr 2025 06:43:14 GMT"} })
126622025-04-30T06:43:14.251ZDEBGtest_snapshot_basic (BackgroundTasks): begin collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
126632025-04-30T06:43:14.251ZDEBGtest_snapshot_basic (BackgroundTasks): end collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
126642025-04-30T06:43:14.252ZDEBGtest_snapshot_basic (BackgroundTasks): finished collection background_task = inventory_collection
126652025-04-30T06:43:14.259ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
126662025-04-30T06:43:14.259ZTRACtest_snapshot_basic (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
126672025-04-30T06:43:14.259ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
126682025-04-30T06:43:14.259ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
126692025-04-30T06:43:14.259ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
126702025-04-30T06:43:14.264ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
126712025-04-30T06:43:14.271ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
126722025-04-30T06:43:14.271ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
126732025-04-30T06:43:14.271ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
126742025-04-30T06:43:14.271ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
126752025-04-30T06:43:14.335ZDEBGtest_snapshot_basic: attempt to fetch latest inventory collection result = Ok("found")
126762025-04-30T06:43:14.340ZWARNtest_snapshot_basic (datastore): Non-retryable transaction failure err = DatabaseError(UniqueViolation, "duplicate key value violates unique constraint \\"inv_caboose_pkey\\"") transaction = inventory_insert_collection
126772025-04-30T06:43:14.340ZWARNtest_snapshot_basic (BackgroundTasks): inventory collection failed background_task = inventory_collection error = failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint "inv_caboose_pkey"
126782025-04-30T06:43:14.340ZDEBGtest_snapshot_basic (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 477.465539ms iteration = 45 status = {"error":"failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint \\"inv_caboose_pkey\\""}
126792025-04-30T06:43:14.340ZDEBGtest_snapshot_basic (BackgroundTasks): activating background_task = inventory_collection iteration = 46 reason = Signaled
126802025-04-30T06:43:14.340ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
126812025-04-30T06:43:14.340ZTRACtest_snapshot_basic (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
126822025-04-30T06:43:14.340ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
126832025-04-30T06:43:14.341ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
126842025-04-30T06:43:14.341ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
126852025-04-30T06:43:14.344ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
126862025-04-30T06:43:14.350ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
126872025-04-30T06:43:14.350ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
126882025-04-30T06:43:14.350ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
126892025-04-30T06:43:14.350ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
126902025-04-30T06:43:14.354ZDEBGtest_snapshot_basic (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = [(d1ad44e5-18dd-4193-a70e-bf9042f0e0e1, 0)]
126912025-04-30T06:43:14.355ZTRACtest_snapshot_basic (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
126922025-04-30T06:43:14.356ZTRACtest_snapshot_basic (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: 6b ac 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 k............_mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:52947 req_id = 221f209f-90a9-4db6-8658-601159f98da8
126932025-04-30T06:43:14.357ZDEBGtest_snapshot_basic (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 27564,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:52947 req_id = 221f209f-90a9-4db6-8658-601159f98da8
126942025-04-30T06:43:14.357ZDEBGtest_snapshot_basic (store): query key key = _mgs._tcp
126952025-04-30T06:43:14.357ZDEBGtest_snapshot_basic (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
126962025-04-30T06:43:14.357ZDEBGtest_snapshot_basic (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:52947 query = LowerQuery { name: LowerName(Name("_mgs._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 41502, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = 221f209f-90a9-4db6-8658-601159f98da8
126972025-04-30T06:43:14.358ZDEBGtest_snapshot_basic (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 41502, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 4259, tv_nsec: 157012530 } })
126982025-04-30T06:43:14.358ZTRACtest_snapshot_basic (DnsResolver): lookup_service_targets: looking up SRV target
126992025-04-30T06:43:14.358ZTRACtest_snapshot_basic (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: 55 fb 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e U...........-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:56569 req_id = 0f695ff8-9325-4137-b32c-1660aff26e75
127002025-04-30T06:43:14.358ZDEBGtest_snapshot_basic (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 22011,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:56569 req_id = 0f695ff8-9325-4137-b32c-1660aff26e75
127012025-04-30T06:43:14.359ZDEBGtest_snapshot_basic (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
127022025-04-30T06:43:14.359ZDEBGtest_snapshot_basic (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:56569 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = 0f695ff8-9325-4137-b32c-1660aff26e75
127032025-04-30T06:43:14.420ZTRACtest_snapshot_basic (DnsResolver): lookup_all_socket_v6 srv dns_name = _clickhouse-admin-keeper._tcp.control-plane.oxide.internal
127042025-04-30T06:43:14.421ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
127052025-04-30T06:43:14.421ZTRACtest_snapshot_basic (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
127062025-04-30T06:43:14.421ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
127072025-04-30T06:43:14.421ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {} }
127082025-04-30T06:43:14.421ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
127092025-04-30T06:43:14.422ZTRACtest_snapshot_basic (dns): buffer buffer = Length: 87 (0x57) bytes\n0000: 8f c2 01 00 00 01 00 00 00 00 00 01 18 5f 63 6c ............._cl\n0010: 69 63 6b 68 6f 75 73 65 2d 61 64 6d 69 6e 2d 6b ickhouse-admin-k\n0020: 65 65 70 65 72 04 5f 74 63 70 0d 63 6f 6e 74 72 eeper._tcp.contr\n0030: 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 69 ol-plane.oxide.i\n0040: 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 04 nternal..!....).\n0050: d0 00 00 00 00 00 00 ....... kind = dns peer_addr = [::1]:63721 req_id = 7d29c607-c0dd-46a2-a97f-8d5c3a6f0c09
127102025-04-30T06:43:14.422ZDEBGtest_snapshot_basic (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 36802,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 24,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 97,\n 100,\n 109,\n 105,\n 110,\n 45,\n 107,\n 101,\n 101,\n 112,\n 101,\n 114,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:63721 req_id = 7d29c607-c0dd-46a2-a97f-8d5c3a6f0c09
127112025-04-30T06:43:14.422ZDEBGtest_snapshot_basic (store): query key key = _clickhouse-admin-keeper._tcp
127122025-04-30T06:43:14.422ZERROtest_snapshot_basic (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 36802, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } NXDOMAIN: no records found for name: "_clickhouse-admin-keeper._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:63721 req_id = 7d29c607-c0dd-46a2-a97f-8d5c3a6f0c09
127132025-04-30T06:43:14.423ZDEBGtest_snapshot_basic (BackgroundTasks): begin collection background_task = inventory_collection
127142025-04-30T06:43:14.423ZDEBGtest_snapshot_basic (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:41502
127152025-04-30T06:43:14.423ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/ignition
127162025-04-30T06:43:14.424ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
127172025-04-30T06:43:14.444ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
127182025-04-30T06:43:14.444ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
127192025-04-30T06:43:14.444ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {} }
127202025-04-30T06:43:14.450ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
127212025-04-30T06:43:14.452ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "d5b67eba-a60f-481f-8e93-71bf539c6496", "content-length": "835", "date": "Wed, 30 Apr 2025 06:43:14 GMT"} })
127222025-04-30T06:43:14.456ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/0
127232025-04-30T06:43:14.462ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
127242025-04-30T06:43:14.462ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {} }
127252025-04-30T06:43:14.465ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
127262025-04-30T06:43:14.467ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "c99cba88-df01-4be9-9aa3-eaca063affb9", "content-length": "734", "date": "Wed, 30 Apr 2025 06:43:14 GMT"} })
127272025-04-30T06:43:14.467ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/0/component/sp/caboose?firmware_slot=0
127282025-04-30T06:43:14.472ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "eddd6292-21b4-4ec1-846d-e754f409a174", "content-length": "111", "date": "Wed, 30 Apr 2025 06:43:14 GMT"} })
127292025-04-30T06:43:14.472ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/0/component/sp/caboose?firmware_slot=1
127302025-04-30T06:43:14.474ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "1e76154a-963a-47ed-bcce-098cc19ee953", "content-length": "111", "date": "Wed, 30 Apr 2025 06:43:14 GMT"} })
127312025-04-30T06:43:14.474ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/0/component/rot/caboose?firmware_slot=0
127322025-04-30T06:43:14.477ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "00473371-64ca-47a3-80f6-ad4da1f8a908", "content-length": "170", "date": "Wed, 30 Apr 2025 06:43:14 GMT"} })
127332025-04-30T06:43:14.477ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/0/component/rot/caboose?firmware_slot=1
127342025-04-30T06:43:14.479ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "7e7a6bfd-8975-44e6-8de1-a19aea964369", "content-length": "170", "date": "Wed, 30 Apr 2025 06:43:14 GMT"} })
127352025-04-30T06:43:14.479ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/0/component/stage0/caboose?firmware_slot=0
127362025-04-30T06:43:14.481ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "13d03466-588f-4edd-9b60-e4638359490f", "content-length": "179", "date": "Wed, 30 Apr 2025 06:43:14 GMT"} })
127372025-04-30T06:43:14.481ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/0/component/stage0/caboose?firmware_slot=1
127382025-04-30T06:43:14.483ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "51f9e96e-0df7-498f-b0a3-326337f17e9d", "content-length": "179", "date": "Wed, 30 Apr 2025 06:43:14 GMT"} })
127392025-04-30T06:43:14.483ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/0/component/rot/cmpa
127402025-04-30T06:43:14.484ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "5db7ace1-820a-4a84-b67f-3817f50de9d2", "content-length": "702", "date": "Wed, 30 Apr 2025 06:43:14 GMT"} })
127412025-04-30T06:43:14.484ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/0/component/rot/cfpa
127422025-04-30T06:43:14.485ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "f5f57770-ecfe-4208-946d-bc126849d250", "content-length": "727", "date": "Wed, 30 Apr 2025 06:43:14 GMT"} })
127432025-04-30T06:43:14.486ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/0/component/rot/cfpa
127442025-04-30T06:43:14.487ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "f6d4d601-d101-491e-adda-c1861e035bfb", "content-length": "729", "date": "Wed, 30 Apr 2025 06:43:14 GMT"} })
127452025-04-30T06:43:14.487ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/0/component/rot/cfpa
127462025-04-30T06:43:14.488ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "9f856c0a-c318-42b6-9185-7f6485e21f71", "content-length": "728", "date": "Wed, 30 Apr 2025 06:43:14 GMT"} })
127472025-04-30T06:43:14.488ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/1
127482025-04-30T06:43:14.489ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "d7740b3d-b519-4924-a246-b7a58ef78d69", "content-length": "734", "date": "Wed, 30 Apr 2025 06:43:14 GMT"} })
127492025-04-30T06:43:14.489ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/1/component/sp/caboose?firmware_slot=0
127502025-04-30T06:43:14.491ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "31af5d6f-cb06-4686-83f4-8d272bd05b4a", "content-length": "111", "date": "Wed, 30 Apr 2025 06:43:14 GMT"} })
127512025-04-30T06:43:14.492ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/1/component/sp/caboose?firmware_slot=1
127522025-04-30T06:43:14.497ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "2bc89cc8-eaec-4bdf-aaff-03bcd6d9bbd0", "content-length": "111", "date": "Wed, 30 Apr 2025 06:43:14 GMT"} })
127532025-04-30T06:43:14.497ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/1/component/rot/caboose?firmware_slot=0
127542025-04-30T06:43:14.497ZDEBGtest_snapshot_basic: attempt to fetch latest inventory collection result = Ok("found")
127552025-04-30T06:43:14.498ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "341c1834-0017-4fe8-97c9-e99de30cf835", "content-length": "170", "date": "Wed, 30 Apr 2025 06:43:14 GMT"} })
127562025-04-30T06:43:14.498ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/1/component/rot/caboose?firmware_slot=1
127572025-04-30T06:43:14.500ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "ec743834-b2b2-48a3-90ef-71e52e37269b", "content-length": "170", "date": "Wed, 30 Apr 2025 06:43:14 GMT"} })
127582025-04-30T06:43:14.501ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/1/component/stage0/caboose?firmware_slot=0
127592025-04-30T06:43:14.503ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "0440dc9b-528e-44bd-b260-1f497e99b00b", "content-length": "179", "date": "Wed, 30 Apr 2025 06:43:14 GMT"} })
127602025-04-30T06:43:14.503ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/1/component/stage0/caboose?firmware_slot=1
127612025-04-30T06:43:14.505ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "42afcce6-41b7-4b9c-a825-9cd935cc1781", "content-length": "179", "date": "Wed, 30 Apr 2025 06:43:14 GMT"} })
127622025-04-30T06:43:14.505ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/1/component/rot/cmpa
127632025-04-30T06:43:14.506ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "440aec39-543e-4e5b-baaf-f515cd1bb49f", "content-length": "702", "date": "Wed, 30 Apr 2025 06:43:14 GMT"} })
127642025-04-30T06:43:14.506ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/1/component/rot/cfpa
127652025-04-30T06:43:14.508ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "7185d2ad-1c35-402c-a994-f7b89aab4aee", "content-length": "727", "date": "Wed, 30 Apr 2025 06:43:14 GMT"} })
127662025-04-30T06:43:14.508ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/1/component/rot/cfpa
127672025-04-30T06:43:14.532ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "40db6a14-0982-4acd-8567-98bce8853b99", "content-length": "729", "date": "Wed, 30 Apr 2025 06:43:14 GMT"} })
127682025-04-30T06:43:14.532ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/1/component/rot/cfpa
127692025-04-30T06:43:14.548ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "a8d443f6-db15-464a-b7d1-ee9151eca717", "content-length": "728", "date": "Wed, 30 Apr 2025 06:43:14 GMT"} })
127702025-04-30T06:43:14.548ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/0
127712025-04-30T06:43:14.549ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
127722025-04-30T06:43:14.550ZTRACtest_snapshot_basic (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
127732025-04-30T06:43:14.550ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
127742025-04-30T06:43:14.550ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {} }
127752025-04-30T06:43:14.553ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
127762025-04-30T06:43:14.587ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
127772025-04-30T06:43:14.598ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
127782025-04-30T06:43:14.598ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
127792025-04-30T06:43:14.598ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {} }
127802025-04-30T06:43:14.600ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
127812025-04-30T06:43:14.618ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
127822025-04-30T06:43:14.618ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {} }
127832025-04-30T06:43:14.619ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
127842025-04-30T06:43:14.627ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "343608a5-e9c0-4605-9a74-d15e37c042f0", "content-length": "723", "date": "Wed, 30 Apr 2025 06:43:14 GMT"} })
127852025-04-30T06:43:14.627ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/0/component/sp/caboose?firmware_slot=0
127862025-04-30T06:43:14.637ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "58c59f54-14bb-4917-89b8-30ca6ffd384b", "content-length": "109", "date": "Wed, 30 Apr 2025 06:43:14 GMT"} })
127872025-04-30T06:43:14.637ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/0/component/sp/caboose?firmware_slot=1
127882025-04-30T06:43:14.640ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "b984faa0-b1f8-49b2-9167-87a25682ebe1", "content-length": "109", "date": "Wed, 30 Apr 2025 06:43:14 GMT"} })
127892025-04-30T06:43:14.640ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/0/component/rot/caboose?firmware_slot=0
127902025-04-30T06:43:14.641ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "3063fe4e-769a-48b0-893f-6b82293b0ae7", "content-length": "169", "date": "Wed, 30 Apr 2025 06:43:14 GMT"} })
127912025-04-30T06:43:14.642ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/0/component/rot/caboose?firmware_slot=1
127922025-04-30T06:43:14.644ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "a258e4f5-85a7-4d1c-8215-f614038b5f32", "content-length": "169", "date": "Wed, 30 Apr 2025 06:43:14 GMT"} })
127932025-04-30T06:43:14.644ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/0/component/stage0/caboose?firmware_slot=0
127942025-04-30T06:43:14.649ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "707cba9c-b224-4837-b036-a322b022231d", "content-length": "178", "date": "Wed, 30 Apr 2025 06:43:14 GMT"} })
127952025-04-30T06:43:14.649ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/0/component/stage0/caboose?firmware_slot=1
127962025-04-30T06:43:14.652ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "3f616468-8f26-4c02-aafa-21c556dc3487", "content-length": "178", "date": "Wed, 30 Apr 2025 06:43:14 GMT"} })
127972025-04-30T06:43:14.652ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/0/component/rot/cmpa
127982025-04-30T06:43:14.654ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "0533d8bd-56b0-4cc1-8dc2-20865a86cf05", "content-length": "702", "date": "Wed, 30 Apr 2025 06:43:14 GMT"} })
127992025-04-30T06:43:14.654ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/0/component/rot/cfpa
128002025-04-30T06:43:14.655ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "12e38f37-da47-431a-aa7f-16b7c80f08f2", "content-length": "727", "date": "Wed, 30 Apr 2025 06:43:14 GMT"} })
128012025-04-30T06:43:14.655ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/0/component/rot/cfpa
128022025-04-30T06:43:14.661ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "dc4805f5-e7b8-4a05-b4a6-a205ea09e053", "content-length": "729", "date": "Wed, 30 Apr 2025 06:43:14 GMT"} })
128032025-04-30T06:43:14.661ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/0/component/rot/cfpa
128042025-04-30T06:43:14.662ZDEBGtest_snapshot_basic: attempt to fetch latest inventory collection result = Ok("found")
128052025-04-30T06:43:14.664ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "4af48fe6-c2e1-4d42-aa99-5196330ca5e5", "content-length": "728", "date": "Wed, 30 Apr 2025 06:43:14 GMT"} })
128062025-04-30T06:43:14.664ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/1
128072025-04-30T06:43:14.665ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "d52c2b4d-4f7d-4210-b06e-534b1ab54b44", "content-length": "723", "date": "Wed, 30 Apr 2025 06:43:14 GMT"} })
128082025-04-30T06:43:14.665ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/1/component/sp/caboose?firmware_slot=0
128092025-04-30T06:43:14.668ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "6b108d41-a9ea-44ec-a3fd-924fb32d5f4a", "content-length": "109", "date": "Wed, 30 Apr 2025 06:43:14 GMT"} })
128102025-04-30T06:43:14.668ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/1/component/sp/caboose?firmware_slot=1
128112025-04-30T06:43:14.670ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "7068fed8-c33d-4884-8b57-2e342c498621", "content-length": "109", "date": "Wed, 30 Apr 2025 06:43:14 GMT"} })
128122025-04-30T06:43:14.671ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/1/component/rot/caboose?firmware_slot=0
128132025-04-30T06:43:14.672ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "5a812460-e136-462f-aeb0-68dde2b79f43", "content-length": "169", "date": "Wed, 30 Apr 2025 06:43:14 GMT"} })
128142025-04-30T06:43:14.672ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/1/component/rot/caboose?firmware_slot=1
128152025-04-30T06:43:14.674ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "43bc25c8-1e1f-49c8-9abb-76059d17bfb9", "content-length": "169", "date": "Wed, 30 Apr 2025 06:43:14 GMT"} })
128162025-04-30T06:43:14.674ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/1/component/stage0/caboose?firmware_slot=0
128172025-04-30T06:43:14.677ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "1db67be2-a456-4260-903a-6d6009439084", "content-length": "178", "date": "Wed, 30 Apr 2025 06:43:14 GMT"} })
128182025-04-30T06:43:14.678ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/1/component/stage0/caboose?firmware_slot=1
128192025-04-30T06:43:14.680ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "97785a85-72f4-446e-a07b-760d9470ce53", "content-length": "178", "date": "Wed, 30 Apr 2025 06:43:14 GMT"} })
128202025-04-30T06:43:14.680ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/1/component/rot/cmpa
128212025-04-30T06:43:14.681ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "14134916-8d4c-4349-9981-7fce7dbc99ae", "content-length": "702", "date": "Wed, 30 Apr 2025 06:43:14 GMT"} })
128222025-04-30T06:43:14.681ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/1/component/rot/cfpa
128232025-04-30T06:43:14.683ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "ec594cd5-f606-4441-af93-25ea97da8838", "content-length": "727", "date": "Wed, 30 Apr 2025 06:43:14 GMT"} })
128242025-04-30T06:43:14.683ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/1/component/rot/cfpa
128252025-04-30T06:43:14.684ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "6c3e750f-9f1a-4bbe-96a8-c967f739e749", "content-length": "729", "date": "Wed, 30 Apr 2025 06:43:14 GMT"} })
128262025-04-30T06:43:14.684ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/1/component/rot/cfpa
128272025-04-30T06:43:14.685ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "63306924-47ab-4ed6-baba-a1e8fd044a24", "content-length": "728", "date": "Wed, 30 Apr 2025 06:43:14 GMT"} })
128282025-04-30T06:43:14.685ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
128292025-04-30T06:43:14.685ZTRACtest_snapshot_basic (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
128302025-04-30T06:43:14.686ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
128312025-04-30T06:43:14.686ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
128322025-04-30T06:43:14.686ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
128332025-04-30T06:43:14.690ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
128342025-04-30T06:43:14.695ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
128352025-04-30T06:43:14.695ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
128362025-04-30T06:43:14.695ZTRACtest_snapshot_basic (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
128372025-04-30T06:43:14.695ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
128382025-04-30T06:43:14.695ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
128392025-04-30T06:43:14.696ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
128402025-04-30T06:43:14.700ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
128412025-04-30T06:43:14.709ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
128422025-04-30T06:43:14.709ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
128432025-04-30T06:43:14.709ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
128442025-04-30T06:43:14.710ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
128452025-04-30T06:43:14.714ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
128462025-04-30T06:43:14.714ZTRACtest_snapshot_basic (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
128472025-04-30T06:43:14.714ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
128482025-04-30T06:43:14.714ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {} }
128492025-04-30T06:43:14.715ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
128502025-04-30T06:43:14.769ZDEBGtest_snapshot_basic (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:41923
128512025-04-30T06:43:14.769ZDEBGtest_snapshot_basic (BackgroundTasks): client request SledAgent = http://[::1]:41923 background_task = inventory_collection body = None method = GET uri = http://[::1]:41923/inventory
128522025-04-30T06:43:14.770ZINFOtest_snapshot_basic (dropshot): accepted connection local_addr = [::1]:41923 remote_addr = [::1]:40767 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
128532025-04-30T06:43:14.770ZTRACtest_snapshot_basic (dropshot): incoming request local_addr = [::1]:41923 method = GET remote_addr = [::1]:40767 req_id = 1ef6feee-d6b6-408f-9cc0-9a079c8a0f45 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
128542025-04-30T06:43:14.771ZINFOtest_snapshot_basic (dropshot): request completed latency_us = 911 local_addr = [::1]:41923 method = GET remote_addr = [::1]:40767 req_id = 1ef6feee-d6b6-408f-9cc0-9a079c8a0f45 response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
128552025-04-30T06:43:14.772ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
128562025-04-30T06:43:14.778ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
128572025-04-30T06:43:14.778ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
128582025-04-30T06:43:14.778ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {} }
128592025-04-30T06:43:14.779ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
128602025-04-30T06:43:14.779ZDEBGtest_snapshot_basic (BackgroundTasks): client response SledAgent = http://[::1]:41923 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:41923/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "1ef6feee-d6b6-408f-9cc0-9a079c8a0f45", "content-length": "3141", "date": "Wed, 30 Apr 2025 06:43:14 GMT"} })
128612025-04-30T06:43:14.782ZDEBGtest_snapshot_basic (BackgroundTasks): begin collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
128622025-04-30T06:43:14.783ZDEBGtest_snapshot_basic (BackgroundTasks): end collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
128632025-04-30T06:43:14.783ZDEBGtest_snapshot_basic (BackgroundTasks): finished collection background_task = inventory_collection
128642025-04-30T06:43:14.787ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
128652025-04-30T06:43:14.788ZTRACtest_snapshot_basic (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
128662025-04-30T06:43:14.788ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
128672025-04-30T06:43:14.788ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
128682025-04-30T06:43:14.790ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
128692025-04-30T06:43:14.793ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
128702025-04-30T06:43:14.793ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {} }
128712025-04-30T06:43:14.794ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
128722025-04-30T06:43:14.795ZDEBGtest_snapshot_basic (collection-task): sent timer-based collection request to the collection task collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
128732025-04-30T06:43:14.795ZDEBGtest_snapshot_basic (collection-task): collecting from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
128742025-04-30T06:43:14.797ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
128752025-04-30T06:43:14.801ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
128762025-04-30T06:43:14.801ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
128772025-04-30T06:43:14.801ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
128782025-04-30T06:43:14.803ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
128792025-04-30T06:43:14.804ZTRACtest_snapshot_basic (collection-task): sent collection request to producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
128802025-04-30T06:43:14.806ZDEBGtest_snapshot_basic (collection-task): collected results from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 n_results = 1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
128812025-04-30T06:43:14.806ZTRACtest_snapshot_basic (collection-task): forwarded results to main collection loop collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
128822025-04-30T06:43:14.806ZTRACtest_snapshot_basic (collection-task): top of inner collection loop, waiting for next request collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
128832025-04-30T06:43:14.837ZWARNtest_snapshot_basic (datastore): Non-retryable transaction failure err = DatabaseError(UniqueViolation, "duplicate key value violates unique constraint \\"inv_caboose_pkey\\"") transaction = inventory_insert_collection
128842025-04-30T06:43:14.837ZWARNtest_snapshot_basic (BackgroundTasks): inventory collection failed background_task = inventory_collection error = failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint "inv_caboose_pkey"
128852025-04-30T06:43:14.837ZDEBGtest_snapshot_basic (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 496.846193ms iteration = 46 status = {"error":"failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint \\"inv_caboose_pkey\\""}
128862025-04-30T06:43:14.837ZDEBGtest_snapshot_basic (BackgroundTasks): activating background_task = inventory_collection iteration = 47 reason = Signaled
128872025-04-30T06:43:14.837ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
128882025-04-30T06:43:14.838ZTRACtest_snapshot_basic (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
128892025-04-30T06:43:14.838ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
128902025-04-30T06:43:14.838ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
128912025-04-30T06:43:14.839ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
128922025-04-30T06:43:14.844ZDEBGtest_snapshot_basic: attempt to fetch latest inventory collection result = Ok("found")
128932025-04-30T06:43:14.845ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
128942025-04-30T06:43:14.852ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
128952025-04-30T06:43:14.852ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
128962025-04-30T06:43:14.852ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
128972025-04-30T06:43:14.852ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
128982025-04-30T06:43:14.856ZDEBGtest_snapshot_basic (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = [(d1ad44e5-18dd-4193-a70e-bf9042f0e0e1, 0)]
128992025-04-30T06:43:14.856ZTRACtest_snapshot_basic (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
129002025-04-30T06:43:14.857ZTRACtest_snapshot_basic (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: 56 d1 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 V............_mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:63943 req_id = 864b5457-8b97-4e07-8b78-8a0502bb7aac
129012025-04-30T06:43:14.857ZDEBGtest_snapshot_basic (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 22225,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:63943 req_id = 864b5457-8b97-4e07-8b78-8a0502bb7aac
129022025-04-30T06:43:14.857ZDEBGtest_snapshot_basic (store): query key key = _mgs._tcp
129032025-04-30T06:43:14.857ZDEBGtest_snapshot_basic (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
129042025-04-30T06:43:14.857ZDEBGtest_snapshot_basic (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:63943 query = LowerQuery { name: LowerName(Name("_mgs._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 41502, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = 864b5457-8b97-4e07-8b78-8a0502bb7aac
129052025-04-30T06:43:14.857ZDEBGtest_snapshot_basic (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 41502, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 4259, tv_nsec: 656809222 } })
129062025-04-30T06:43:14.858ZTRACtest_snapshot_basic (DnsResolver): lookup_service_targets: looking up SRV target
129072025-04-30T06:43:14.858ZTRACtest_snapshot_basic (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: ea 7c 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e .|..........-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:59940 req_id = add49152-376d-4a26-acc0-766762867aa3
129082025-04-30T06:43:14.858ZDEBGtest_snapshot_basic (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 60028,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:59940 req_id = add49152-376d-4a26-acc0-766762867aa3
129092025-04-30T06:43:14.858ZDEBGtest_snapshot_basic (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
129102025-04-30T06:43:14.858ZDEBGtest_snapshot_basic (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:59940 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = add49152-376d-4a26-acc0-766762867aa3
129112025-04-30T06:43:14.921ZTRACtest_snapshot_basic (DnsResolver): lookup_all_socket_v6 srv dns_name = _clickhouse-admin-keeper._tcp.control-plane.oxide.internal
129122025-04-30T06:43:14.922ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
129132025-04-30T06:43:14.922ZTRACtest_snapshot_basic (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
129142025-04-30T06:43:14.922ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
129152025-04-30T06:43:14.922ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {} }
129162025-04-30T06:43:14.923ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
129172025-04-30T06:43:14.923ZTRACtest_snapshot_basic (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal
129182025-04-30T06:43:14.924ZTRACtest_snapshot_basic (dns): buffer buffer = Length: 87 (0x57) bytes\n0000: 1b cc 01 00 00 01 00 00 00 00 00 01 18 5f 63 6c ............._cl\n0010: 69 63 6b 68 6f 75 73 65 2d 61 64 6d 69 6e 2d 6b ickhouse-admin-k\n0020: 65 65 70 65 72 04 5f 74 63 70 0d 63 6f 6e 74 72 eeper._tcp.contr\n0030: 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 69 ol-plane.oxide.i\n0040: 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 04 nternal..!....).\n0050: d0 00 00 00 00 00 00 ....... kind = dns peer_addr = [::1]:64983 req_id = 9b760b7d-4dca-4acd-b139-1771ce0b18fa
129192025-04-30T06:43:14.924ZDEBGtest_snapshot_basic (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 7116,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 24,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 97,\n 100,\n 109,\n 105,\n 110,\n 45,\n 107,\n 101,\n 101,\n 112,\n 101,\n 114,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:64983 req_id = 9b760b7d-4dca-4acd-b139-1771ce0b18fa
129202025-04-30T06:43:14.924ZDEBGtest_snapshot_basic (store): query key key = _clickhouse-admin-keeper._tcp
129212025-04-30T06:43:14.925ZERROtest_snapshot_basic (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 7116, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } NXDOMAIN: no records found for name: "_clickhouse-admin-keeper._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:64983 req_id = 9b760b7d-4dca-4acd-b139-1771ce0b18fa
129222025-04-30T06:43:14.925ZDEBGtest_snapshot_basic (BackgroundTasks): begin collection background_task = inventory_collection
129232025-04-30T06:43:14.925ZDEBGtest_snapshot_basic (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:41502
129242025-04-30T06:43:14.925ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/ignition
129252025-04-30T06:43:14.925ZTRACtest_snapshot_basic (dns): buffer buffer = Length: 72 (0x48) bytes\n0000: c5 89 01 00 00 01 00 00 00 00 00 01 09 5f 64 65 ............._de\n0010: 6e 64 72 69 74 65 04 5f 74 63 70 0d 63 6f 6e 74 ndrite._tcp.cont\n0020: 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 rol-plane.oxide.\n0030: 69 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 internal..!....)\n0040: 04 d0 00 00 00 00 00 00 ........ kind = dns peer_addr = [::1]:63599 req_id = 744f14d0-6eda-40cd-9041-6d8b4fcfb205
129262025-04-30T06:43:14.926ZDEBGtest_snapshot_basic (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 50569,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_dendrite._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_dendrite._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 9,\n 95,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:63599 req_id = 744f14d0-6eda-40cd-9041-6d8b4fcfb205
129272025-04-30T06:43:14.932ZDEBGtest_snapshot_basic (store): query key key = _dendrite._tcp
129282025-04-30T06:43:14.933ZDEBGtest_snapshot_basic (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
129292025-04-30T06:43:14.933ZDEBGtest_snapshot_basic (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:63599 query = LowerQuery { name: LowerName(Name("_dendrite._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 32976, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = 744f14d0-6eda-40cd-9041-6d8b4fcfb205
129302025-04-30T06:43:14.934ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
129312025-04-30T06:43:14.939ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
129322025-04-30T06:43:14.939ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
129332025-04-30T06:43:14.939ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {} }
129342025-04-30T06:43:14.940ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
129352025-04-30T06:43:14.941ZDEBGtest_snapshot_basic (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 32976, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 4259, tv_nsec: 740297908 } })
129362025-04-30T06:43:14.941ZTRACtest_snapshot_basic (DnsResolver): lookup_service_targets: looking up SRV target
129372025-04-30T06:43:14.942ZTRACtest_snapshot_basic (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal
129382025-04-30T06:43:14.944ZTRACtest_snapshot_basic (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: 61 a9 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e a...........-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:49554 req_id = 92cd6c44-59a3-4378-a8f8-17e8ac1b68da
129392025-04-30T06:43:14.944ZDEBGtest_snapshot_basic (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 25001,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:49554 req_id = 92cd6c44-59a3-4378-a8f8-17e8ac1b68da
129402025-04-30T06:43:14.944ZDEBGtest_snapshot_basic (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
129412025-04-30T06:43:14.945ZDEBGtest_snapshot_basic (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:49554 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = 92cd6c44-59a3-4378-a8f8-17e8ac1b68da
129422025-04-30T06:43:14.946ZINFOtest_snapshot_basic (BackgroundTasks): Determining switch slots managed by switch zones background_task = nat_v4_garbage_collector
129432025-04-30T06:43:15.038ZINFOtest_snapshot_basic (BackgroundTasks): determining switch slot managed by dendrite zone background_task = nat_v4_garbage_collector zone_address = ::1
129442025-04-30T06:43:15.038ZDEBGtest_snapshot_basic (MgsClient): client request background_task = nat_v4_garbage_collector body = None method = GET uri = http://[::1]:12225/local/switch-id
129452025-04-30T06:43:15.038ZDEBGtest_snapshot_basic (MgsClient): client response background_task = nat_v4_garbage_collector result = Err(reqwest::Error { kind: Request, url: "http://[::1]:12225/local/switch-id", source: hyper_util::client::legacy::Error(Connect, ConnectError("tcp connect error", Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) })
129462025-04-30T06:43:15.038ZWARNtest_snapshot_basic (BackgroundTasks): failed to identify switch slot for dendrite background_task = nat_v4_garbage_collector reason = Communication Error: error sending request for url (http://[::1]:12225/local/switch-id): client error (Connect): tcp connect error: Connection refused (os error 146): Connection refused (os error 146) zone_address = ::1
129472025-04-30T06:43:15.040ZWARNtest_snapshot_basic (BackgroundTasks): Failed to map switch zone addr: Communication Error: error sending request for url (http://[::1]:12225/local/switch-id), retrying background_task = nat_v4_garbage_collector
129482025-04-30T06:43:15.044ZTRACtest_snapshot_basic (dns): buffer buffer = Length: 72 (0x48) bytes\n0000: bd 95 01 00 00 01 00 00 00 00 00 01 09 5f 64 65 ............._de\n0010: 6e 64 72 69 74 65 04 5f 74 63 70 0d 63 6f 6e 74 ndrite._tcp.cont\n0020: 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 rol-plane.oxide.\n0030: 69 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 internal..!....)\n0040: 04 d0 00 00 00 00 00 00 ........ kind = dns peer_addr = [::1]:53488 req_id = 66038b2b-2334-4ca4-853d-b029967c6a34
129492025-04-30T06:43:15.045ZDEBGtest_snapshot_basic (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 48533,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_dendrite._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_dendrite._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 9,\n 95,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:53488 req_id = 66038b2b-2334-4ca4-853d-b029967c6a34
129502025-04-30T06:43:15.046ZDEBGtest_snapshot_basic (store): query key key = _dendrite._tcp
129512025-04-30T06:43:15.046ZDEBGtest_snapshot_basic (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
129522025-04-30T06:43:15.046ZDEBGtest_snapshot_basic (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:53488 query = LowerQuery { name: LowerName(Name("_dendrite._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 32976, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = 66038b2b-2334-4ca4-853d-b029967c6a34
129532025-04-30T06:43:15.048ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
129542025-04-30T06:43:15.048ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {} }
129552025-04-30T06:43:15.050ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
129562025-04-30T06:43:15.052ZDEBGtest_snapshot_basic (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 32976, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 4259, tv_nsec: 850051905 } })
129572025-04-30T06:43:15.052ZTRACtest_snapshot_basic (DnsResolver): lookup_service_targets: looking up SRV target
129582025-04-30T06:43:15.052ZTRACtest_snapshot_basic (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal
129592025-04-30T06:43:15.052ZDEBGtest_snapshot_basic (results-sink): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
129602025-04-30T06:43:15.052ZDEBGtest_snapshot_basic (results-sink-cluster): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
129612025-04-30T06:43:15.054ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "6be57cd2-670b-4bb3-8eac-0debc4500294", "content-length": "835", "date": "Wed, 30 Apr 2025 06:43:14 GMT"} })
129622025-04-30T06:43:15.054ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/0
129632025-04-30T06:43:15.063ZDEBGtest_snapshot_basic (clickhouse-client): unrolling 1 total samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 6ad06c6b-9a6e-4ed1-b3f4-5d2e9da3f8a7
129642025-04-30T06:43:15.065ZTRACtest_snapshot_basic (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 6ad06c6b-9a6e-4ed1-b3f4-5d2e9da3f8a7 n_columns = 4 n_rows = 2 sql = INSERT INTO oximeter.fields_string FORMAT Native
129652025-04-30T06:43:15.067ZTRACtest_snapshot_basic (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: be b7 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e ............-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:63109 req_id = 0de40a6f-1fc3-484a-8c4d-3f420c20abbf
129662025-04-30T06:43:15.067ZDEBGtest_snapshot_basic (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 48823,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:63109 req_id = 0de40a6f-1fc3-484a-8c4d-3f420c20abbf
129672025-04-30T06:43:15.068ZDEBGtest_snapshot_basic (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
129682025-04-30T06:43:15.068ZDEBGtest_snapshot_basic (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:63109 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = 0de40a6f-1fc3-484a-8c4d-3f420c20abbf
129692025-04-30T06:43:15.068ZTRACtest_snapshot_basic (dns): buffer buffer = Length: 72 (0x48) bytes\n0000: af 61 01 00 00 01 00 00 00 00 00 01 09 5f 64 65 .a..........._de\n0010: 6e 64 72 69 74 65 04 5f 74 63 70 0d 63 6f 6e 74 ndrite._tcp.cont\n0020: 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 rol-plane.oxide.\n0030: 69 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 internal..!....)\n0040: 04 d0 00 00 00 00 00 00 ........ kind = dns peer_addr = [::1]:62682 req_id = 74f55d93-0d1b-4133-abc8-c0c1fd42eec5
129702025-04-30T06:43:15.068ZDEBGtest_snapshot_basic (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 44897,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_dendrite._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_dendrite._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 9,\n 95,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:62682 req_id = 74f55d93-0d1b-4133-abc8-c0c1fd42eec5
129712025-04-30T06:43:15.068ZDEBGtest_snapshot_basic (store): query key key = _dendrite._tcp
129722025-04-30T06:43:15.068ZDEBGtest_snapshot_basic (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
129732025-04-30T06:43:15.069ZDEBGtest_snapshot_basic (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:62682 query = LowerQuery { name: LowerName(Name("_dendrite._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 32976, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = 74f55d93-0d1b-4133-abc8-c0c1fd42eec5
129742025-04-30T06:43:15.072ZINFOtest_snapshot_basic (BackgroundTasks): Determining switch slots managed by switch zones background_task = bfd_manager
129752025-04-30T06:43:15.161ZINFOtest_snapshot_basic (BackgroundTasks): determining switch slot managed by dendrite zone background_task = bfd_manager zone_address = ::1
129762025-04-30T06:43:15.161ZDEBGtest_snapshot_basic (MgsClient): client request background_task = bfd_manager body = None method = GET uri = http://[::1]:12225/local/switch-id
129772025-04-30T06:43:15.162ZDEBGtest_snapshot_basic (MgsClient): client response background_task = bfd_manager result = Err(reqwest::Error { kind: Request, url: "http://[::1]:12225/local/switch-id", source: hyper_util::client::legacy::Error(Connect, ConnectError("tcp connect error", Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) })
129782025-04-30T06:43:15.162ZWARNtest_snapshot_basic (BackgroundTasks): failed to identify switch slot for dendrite background_task = bfd_manager reason = Communication Error: error sending request for url (http://[::1]:12225/local/switch-id): client error (Connect): tcp connect error: Connection refused (os error 146): Connection refused (os error 146) zone_address = ::1
129792025-04-30T06:43:15.164ZWARNtest_snapshot_basic (BackgroundTasks): Failed to map switch zone addr: Communication Error: error sending request for url (http://[::1]:12225/local/switch-id), retrying background_task = bfd_manager
129802025-04-30T06:43:15.165ZDEBGtest_snapshot_basic (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 6ad06c6b-9a6e-4ed1-b3f4-5d2e9da3f8a7 n_rows = 2 table_name = fields_string
129812025-04-30T06:43:15.165ZTRACtest_snapshot_basic (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 6ad06c6b-9a6e-4ed1-b3f4-5d2e9da3f8a7 n_columns = 4 n_rows = 1 sql = INSERT INTO oximeter.measurements_i64 FORMAT Native
129822025-04-30T06:43:15.166ZDEBGtest_snapshot_basic (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 32976, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 4259, tv_nsec: 965646360 } })
129832025-04-30T06:43:15.166ZTRACtest_snapshot_basic (DnsResolver): lookup_service_targets: looking up SRV target
129842025-04-30T06:43:15.169ZWARNtest_snapshot_basic (results-sink-cluster): failed to insert some results into metric DB collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 error = Failed to check out connection to database: No backends found for this service file = oximeter/collector/src/results_sink.rs:92
129852025-04-30T06:43:15.176ZTRACtest_snapshot_basic (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: 7b 49 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e {I..........-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:64691 req_id = 0ac84a3d-dd4f-4157-be89-013b2c39924a
129862025-04-30T06:43:15.176ZDEBGtest_snapshot_basic (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 31561,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:64691 req_id = 0ac84a3d-dd4f-4157-be89-013b2c39924a
129872025-04-30T06:43:15.176ZDEBGtest_snapshot_basic (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
129882025-04-30T06:43:15.176ZDEBGtest_snapshot_basic (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:64691 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = 0ac84a3d-dd4f-4157-be89-013b2c39924a
129892025-04-30T06:43:15.176ZINFOtest_snapshot_basic (BackgroundTasks): Determining switch slots managed by switch zones background_task = switch_port_config_manager rack_id = c19a698f-c6f9-4a17-ae30-20d711b8f7dc
129902025-04-30T06:43:15.244ZINFOtest_snapshot_basic (BackgroundTasks): determining switch slot managed by dendrite zone background_task = switch_port_config_manager rack_id = c19a698f-c6f9-4a17-ae30-20d711b8f7dc zone_address = ::1
129912025-04-30T06:43:15.244ZDEBGtest_snapshot_basic (MgsClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = c19a698f-c6f9-4a17-ae30-20d711b8f7dc uri = http://[::1]:12225/local/switch-id
129922025-04-30T06:43:15.244ZDEBGtest_snapshot_basic (MgsClient): client response background_task = switch_port_config_manager rack_id = c19a698f-c6f9-4a17-ae30-20d711b8f7dc result = Err(reqwest::Error { kind: Request, url: "http://[::1]:12225/local/switch-id", source: hyper_util::client::legacy::Error(Connect, ConnectError("tcp connect error", Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) })
129932025-04-30T06:43:15.244ZWARNtest_snapshot_basic (BackgroundTasks): failed to identify switch slot for dendrite background_task = switch_port_config_manager rack_id = c19a698f-c6f9-4a17-ae30-20d711b8f7dc reason = Communication Error: error sending request for url (http://[::1]:12225/local/switch-id): client error (Connect): tcp connect error: Connection refused (os error 146): Connection refused (os error 146) zone_address = ::1
129942025-04-30T06:43:15.246ZWARNtest_snapshot_basic (BackgroundTasks): Failed to map switch zone addr: Communication Error: error sending request for url (http://[::1]:12225/local/switch-id), retrying background_task = switch_port_config_manager rack_id = c19a698f-c6f9-4a17-ae30-20d711b8f7dc
129952025-04-30T06:43:15.246ZDEBGtest_snapshot_basic (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 6ad06c6b-9a6e-4ed1-b3f4-5d2e9da3f8a7 n_rows = 1 table_name = measurements_i64
129962025-04-30T06:43:15.246ZTRACtest_snapshot_basic (results-sink): successfully inserted samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
129972025-04-30T06:43:15.246ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "f67f67f0-2c94-4401-b808-3c7887cfe2f9", "content-length": "734", "date": "Wed, 30 Apr 2025 06:43:14 GMT"} })
129982025-04-30T06:43:15.246ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/0/component/sp/caboose?firmware_slot=0
129992025-04-30T06:43:15.249ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "dfad0ac4-4a95-49d0-ba57-e16f413b4d50", "content-length": "111", "date": "Wed, 30 Apr 2025 06:43:15 GMT"} })
130002025-04-30T06:43:15.249ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/0/component/sp/caboose?firmware_slot=1
130012025-04-30T06:43:15.252ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "37b62aee-336c-458f-b9ce-40b9d256e47e", "content-length": "111", "date": "Wed, 30 Apr 2025 06:43:15 GMT"} })
130022025-04-30T06:43:15.252ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/0/component/rot/caboose?firmware_slot=0
130032025-04-30T06:43:15.255ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "945069bb-4401-49c7-8b86-475103df5fb4", "content-length": "170", "date": "Wed, 30 Apr 2025 06:43:15 GMT"} })
130042025-04-30T06:43:15.255ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/0/component/rot/caboose?firmware_slot=1
130052025-04-30T06:43:15.261ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "db319d8c-d300-4c67-94d0-2657999c81c6", "content-length": "170", "date": "Wed, 30 Apr 2025 06:43:15 GMT"} })
130062025-04-30T06:43:15.261ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/0/component/stage0/caboose?firmware_slot=0
130072025-04-30T06:43:15.264ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "9a4c70de-51be-43d0-a1fb-6af1f3c53c2a", "content-length": "179", "date": "Wed, 30 Apr 2025 06:43:15 GMT"} })
130082025-04-30T06:43:15.264ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/0/component/stage0/caboose?firmware_slot=1
130092025-04-30T06:43:15.272ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "393024df-00b1-43b4-a689-38b4bd082c61", "content-length": "179", "date": "Wed, 30 Apr 2025 06:43:15 GMT"} })
130102025-04-30T06:43:15.272ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/0/component/rot/cmpa
130112025-04-30T06:43:15.273ZDEBGtest_snapshot_basic: attempt to fetch latest inventory collection result = Ok("found")
130122025-04-30T06:43:15.277ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "a3e5ea65-c633-4883-b55d-f830012f9748", "content-length": "702", "date": "Wed, 30 Apr 2025 06:43:15 GMT"} })
130132025-04-30T06:43:15.277ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/0/component/rot/cfpa
130142025-04-30T06:43:15.278ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "70c1dc3a-b2e9-4123-bae3-6851d9f15ee9", "content-length": "727", "date": "Wed, 30 Apr 2025 06:43:15 GMT"} })
130152025-04-30T06:43:15.278ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/0/component/rot/cfpa
130162025-04-30T06:43:15.280ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "25ca10cc-c361-4634-bc37-507eb4755a45", "content-length": "729", "date": "Wed, 30 Apr 2025 06:43:15 GMT"} })
130172025-04-30T06:43:15.280ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/0/component/rot/cfpa
130182025-04-30T06:43:15.282ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "e064d850-eef2-451a-bb26-0b8ebb77346b", "content-length": "728", "date": "Wed, 30 Apr 2025 06:43:15 GMT"} })
130192025-04-30T06:43:15.283ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/1
130202025-04-30T06:43:15.287ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "7183b186-30b2-4c2a-8e17-5c61f4b83b0f", "content-length": "734", "date": "Wed, 30 Apr 2025 06:43:15 GMT"} })
130212025-04-30T06:43:15.287ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/1/component/sp/caboose?firmware_slot=0
130222025-04-30T06:43:15.290ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "25f3e539-f6e7-4818-a576-c34392433a00", "content-length": "111", "date": "Wed, 30 Apr 2025 06:43:15 GMT"} })
130232025-04-30T06:43:15.290ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/1/component/sp/caboose?firmware_slot=1
130242025-04-30T06:43:15.293ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "b62a468b-96ed-4524-90ca-76bde82c2999", "content-length": "111", "date": "Wed, 30 Apr 2025 06:43:15 GMT"} })
130252025-04-30T06:43:15.293ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/1/component/rot/caboose?firmware_slot=0
130262025-04-30T06:43:15.297ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "b8327fa1-ee03-4b1a-a970-d67ffb579eae", "content-length": "170", "date": "Wed, 30 Apr 2025 06:43:15 GMT"} })
130272025-04-30T06:43:15.297ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/1/component/rot/caboose?firmware_slot=1
130282025-04-30T06:43:15.300ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "bc244624-1b0e-4948-835c-6ba884ab507d", "content-length": "170", "date": "Wed, 30 Apr 2025 06:43:15 GMT"} })
130292025-04-30T06:43:15.300ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/1/component/stage0/caboose?firmware_slot=0
130302025-04-30T06:43:15.305ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "2d3b4f44-a913-4bb1-94a4-da7adab2ad96", "content-length": "179", "date": "Wed, 30 Apr 2025 06:43:15 GMT"} })
130312025-04-30T06:43:15.305ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/1/component/stage0/caboose?firmware_slot=1
130322025-04-30T06:43:15.309ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "9d047d5b-f04d-4cfe-b505-14f7c860a68f", "content-length": "179", "date": "Wed, 30 Apr 2025 06:43:15 GMT"} })
130332025-04-30T06:43:15.311ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/1/component/rot/cmpa
130342025-04-30T06:43:15.316ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "2641fe95-8fd0-49a9-9903-1679f3ca33dd", "content-length": "702", "date": "Wed, 30 Apr 2025 06:43:15 GMT"} })
130352025-04-30T06:43:15.316ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/1/component/rot/cfpa
130362025-04-30T06:43:15.319ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "289c82a0-e535-4701-bdc8-94ccfb4a6123", "content-length": "727", "date": "Wed, 30 Apr 2025 06:43:15 GMT"} })
130372025-04-30T06:43:15.320ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/1/component/rot/cfpa
130382025-04-30T06:43:15.320ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "d3a4b9f1-3bf3-46f7-a156-ecc027c03fe3", "content-length": "729", "date": "Wed, 30 Apr 2025 06:43:15 GMT"} })
130392025-04-30T06:43:15.321ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/1/component/rot/cfpa
130402025-04-30T06:43:15.324ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "a286e5af-c88a-41c3-9662-50975195d2e2", "content-length": "728", "date": "Wed, 30 Apr 2025 06:43:15 GMT"} })
130412025-04-30T06:43:15.324ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/0
130422025-04-30T06:43:15.326ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
130432025-04-30T06:43:15.326ZTRACtest_snapshot_basic (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
130442025-04-30T06:43:15.326ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
130452025-04-30T06:43:15.326ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {} }
130462025-04-30T06:43:15.327ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
130472025-04-30T06:43:15.328ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "b12d03ff-1a7e-409b-a554-8750a99e2588", "content-length": "723", "date": "Wed, 30 Apr 2025 06:43:15 GMT"} })
130482025-04-30T06:43:15.329ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/0/component/sp/caboose?firmware_slot=0
130492025-04-30T06:43:15.330ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
130502025-04-30T06:43:15.336ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
130512025-04-30T06:43:15.336ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
130522025-04-30T06:43:15.337ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {} }
130532025-04-30T06:43:15.338ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
130542025-04-30T06:43:15.341ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "2e8f72a9-a516-4ac8-afd7-b91cd48efa86", "content-length": "109", "date": "Wed, 30 Apr 2025 06:43:15 GMT"} })
130552025-04-30T06:43:15.341ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/0/component/sp/caboose?firmware_slot=1
130562025-04-30T06:43:15.342ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
130572025-04-30T06:43:15.342ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {} }
130582025-04-30T06:43:15.343ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
130592025-04-30T06:43:15.345ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "a11b1b3a-352a-4322-8a66-cfb9a7335210", "content-length": "109", "date": "Wed, 30 Apr 2025 06:43:15 GMT"} })
130602025-04-30T06:43:15.345ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/0/component/rot/caboose?firmware_slot=0
130612025-04-30T06:43:15.347ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "1b85e189-279d-4b99-ad52-b8aaec72df08", "content-length": "169", "date": "Wed, 30 Apr 2025 06:43:15 GMT"} })
130622025-04-30T06:43:15.348ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/0/component/rot/caboose?firmware_slot=1
130632025-04-30T06:43:15.351ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "e263ec0d-713f-4228-91fe-64a1177877c6", "content-length": "169", "date": "Wed, 30 Apr 2025 06:43:15 GMT"} })
130642025-04-30T06:43:15.351ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/0/component/stage0/caboose?firmware_slot=0
130652025-04-30T06:43:15.354ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "4c4bae3d-20ff-4b34-8462-a8ada8f4e040", "content-length": "178", "date": "Wed, 30 Apr 2025 06:43:15 GMT"} })
130662025-04-30T06:43:15.354ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/0/component/stage0/caboose?firmware_slot=1
130672025-04-30T06:43:15.356ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "f15ac60d-b1bf-4edf-a4e1-35f5b1a7b440", "content-length": "178", "date": "Wed, 30 Apr 2025 06:43:15 GMT"} })
130682025-04-30T06:43:15.356ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/0/component/rot/cmpa
130692025-04-30T06:43:15.357ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "cd722630-abb2-4224-ab29-7b696fd062ec", "content-length": "702", "date": "Wed, 30 Apr 2025 06:43:15 GMT"} })
130702025-04-30T06:43:15.358ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/0/component/rot/cfpa
130712025-04-30T06:43:15.361ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "8af9bfd1-638f-4128-b541-f25935789c41", "content-length": "727", "date": "Wed, 30 Apr 2025 06:43:15 GMT"} })
130722025-04-30T06:43:15.361ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/0/component/rot/cfpa
130732025-04-30T06:43:15.363ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "752c2d7d-25f2-4efb-a0e3-6cafd903a784", "content-length": "729", "date": "Wed, 30 Apr 2025 06:43:15 GMT"} })
130742025-04-30T06:43:15.363ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/0/component/rot/cfpa
130752025-04-30T06:43:15.364ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "97cfa8f8-6f54-41ef-a45f-5fdc98a0acf5", "content-length": "728", "date": "Wed, 30 Apr 2025 06:43:15 GMT"} })
130762025-04-30T06:43:15.366ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/1
130772025-04-30T06:43:15.367ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "b1410916-d3ed-4286-b29f-94e6e2145862", "content-length": "723", "date": "Wed, 30 Apr 2025 06:43:15 GMT"} })
130782025-04-30T06:43:15.367ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/1/component/sp/caboose?firmware_slot=0
130792025-04-30T06:43:15.369ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "dbfd3024-ed62-4a24-89f3-98f4783fe0fe", "content-length": "109", "date": "Wed, 30 Apr 2025 06:43:15 GMT"} })
130802025-04-30T06:43:15.369ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/1/component/sp/caboose?firmware_slot=1
130812025-04-30T06:43:15.373ZDEBGtest_snapshot_basic: attempt to fetch latest inventory collection result = Ok("found")
130822025-04-30T06:43:15.375ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "ed4ffa6b-9c47-4b9b-b55a-95ae743b6529", "content-length": "109", "date": "Wed, 30 Apr 2025 06:43:15 GMT"} })
130832025-04-30T06:43:15.376ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/1/component/rot/caboose?firmware_slot=0
130842025-04-30T06:43:15.378ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "3d84276c-997e-4ded-afd0-e30ce3e9d8f6", "content-length": "169", "date": "Wed, 30 Apr 2025 06:43:15 GMT"} })
130852025-04-30T06:43:15.378ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/1/component/rot/caboose?firmware_slot=1
130862025-04-30T06:43:15.380ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "0f6a5405-954f-4da0-8cd9-7e094d829241", "content-length": "169", "date": "Wed, 30 Apr 2025 06:43:15 GMT"} })
130872025-04-30T06:43:15.380ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/1/component/stage0/caboose?firmware_slot=0
130882025-04-30T06:43:15.382ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "49cdf9b4-b649-4357-97dd-b0dd45fbb636", "content-length": "178", "date": "Wed, 30 Apr 2025 06:43:15 GMT"} })
130892025-04-30T06:43:15.382ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/1/component/stage0/caboose?firmware_slot=1
130902025-04-30T06:43:15.384ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "35ab7f1d-a3f1-4e8c-bdc7-8c112927fc83", "content-length": "178", "date": "Wed, 30 Apr 2025 06:43:15 GMT"} })
130912025-04-30T06:43:15.384ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/1/component/rot/cmpa
130922025-04-30T06:43:15.385ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "c7e128c3-37d6-4ee8-90c4-ad7d84fd0607", "content-length": "702", "date": "Wed, 30 Apr 2025 06:43:15 GMT"} })
130932025-04-30T06:43:15.385ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/1/component/rot/cfpa
130942025-04-30T06:43:15.386ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "b8ee6d18-d8f0-472a-8852-dd2fafab8c7f", "content-length": "727", "date": "Wed, 30 Apr 2025 06:43:15 GMT"} })
130952025-04-30T06:43:15.386ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/1/component/rot/cfpa
130962025-04-30T06:43:15.388ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "e2ef974e-d2f1-4810-93b8-44e9bf6b2353", "content-length": "729", "date": "Wed, 30 Apr 2025 06:43:15 GMT"} })
130972025-04-30T06:43:15.393ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/1/component/rot/cfpa
130982025-04-30T06:43:15.394ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "eba38c58-be4e-4b02-a615-07d32b0e1c3a", "content-length": "728", "date": "Wed, 30 Apr 2025 06:43:15 GMT"} })
130992025-04-30T06:43:15.394ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
131002025-04-30T06:43:15.394ZTRACtest_snapshot_basic (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
131012025-04-30T06:43:15.394ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
131022025-04-30T06:43:15.394ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
131032025-04-30T06:43:15.395ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
131042025-04-30T06:43:15.399ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
131052025-04-30T06:43:15.407ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
131062025-04-30T06:43:15.407ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
131072025-04-30T06:43:15.407ZTRACtest_snapshot_basic (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
131082025-04-30T06:43:15.407ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
131092025-04-30T06:43:15.408ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
131102025-04-30T06:43:15.410ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
131112025-04-30T06:43:15.414ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
131122025-04-30T06:43:15.420ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
131132025-04-30T06:43:15.420ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
131142025-04-30T06:43:15.420ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
131152025-04-30T06:43:15.421ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
131162025-04-30T06:43:15.426ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
131172025-04-30T06:43:15.426ZTRACtest_snapshot_basic (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
131182025-04-30T06:43:15.426ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
131192025-04-30T06:43:15.426ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {} }
131202025-04-30T06:43:15.427ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
131212025-04-30T06:43:15.512ZDEBGtest_snapshot_basic (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:41923
131222025-04-30T06:43:15.513ZDEBGtest_snapshot_basic (BackgroundTasks): client request SledAgent = http://[::1]:41923 background_task = inventory_collection body = None method = GET uri = http://[::1]:41923/inventory
131232025-04-30T06:43:15.513ZINFOtest_snapshot_basic (dropshot): accepted connection local_addr = [::1]:41923 remote_addr = [::1]:37898 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
131242025-04-30T06:43:15.522ZTRACtest_snapshot_basic (dropshot): incoming request local_addr = [::1]:41923 method = GET remote_addr = [::1]:37898 req_id = edcde3a4-23c5-4f51-9d43-1f31abda8fe2 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
131252025-04-30T06:43:15.523ZINFOtest_snapshot_basic (dropshot): request completed latency_us = 849 local_addr = [::1]:41923 method = GET remote_addr = [::1]:37898 req_id = edcde3a4-23c5-4f51-9d43-1f31abda8fe2 response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
131262025-04-30T06:43:15.523ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
131272025-04-30T06:43:15.532ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
131282025-04-30T06:43:15.533ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
131292025-04-30T06:43:15.533ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {} }
131302025-04-30T06:43:15.534ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
131312025-04-30T06:43:15.535ZDEBGtest_snapshot_basic (BackgroundTasks): client response SledAgent = http://[::1]:41923 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:41923/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "edcde3a4-23c5-4f51-9d43-1f31abda8fe2", "content-length": "3141", "date": "Wed, 30 Apr 2025 06:43:15 GMT"} })
131322025-04-30T06:43:15.538ZDEBGtest_snapshot_basic (BackgroundTasks): begin collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
131332025-04-30T06:43:15.538ZDEBGtest_snapshot_basic (BackgroundTasks): end collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
131342025-04-30T06:43:15.538ZDEBGtest_snapshot_basic (BackgroundTasks): finished collection background_task = inventory_collection
131352025-04-30T06:43:15.540ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
131362025-04-30T06:43:15.541ZTRACtest_snapshot_basic (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
131372025-04-30T06:43:15.541ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
131382025-04-30T06:43:15.541ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
131392025-04-30T06:43:15.541ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
131402025-04-30T06:43:15.550ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
131412025-04-30T06:43:15.550ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {} }
131422025-04-30T06:43:15.553ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
131432025-04-30T06:43:15.553ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
131442025-04-30T06:43:15.558ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
131452025-04-30T06:43:15.561ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
131462025-04-30T06:43:15.561ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
131472025-04-30T06:43:15.563ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
131482025-04-30T06:43:15.640ZWARNtest_snapshot_basic (datastore): Non-retryable transaction failure err = DatabaseError(UniqueViolation, "duplicate key value violates unique constraint \\"inv_caboose_pkey\\"") transaction = inventory_insert_collection
131492025-04-30T06:43:15.641ZWARNtest_snapshot_basic (BackgroundTasks): inventory collection failed background_task = inventory_collection error = failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint "inv_caboose_pkey"
131502025-04-30T06:43:15.646ZDEBGtest_snapshot_basic (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 805.038487ms iteration = 47 status = {"error":"failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint \\"inv_caboose_pkey\\""}
131512025-04-30T06:43:15.648ZDEBGtest_snapshot_basic (BackgroundTasks): activating background_task = inventory_collection iteration = 48 reason = Signaled
131522025-04-30T06:43:15.648ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
131532025-04-30T06:43:15.649ZTRACtest_snapshot_basic (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
131542025-04-30T06:43:15.649ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
131552025-04-30T06:43:15.649ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
131562025-04-30T06:43:15.650ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
131572025-04-30T06:43:15.655ZDEBGtest_snapshot_basic: attempt to fetch latest inventory collection result = Ok("found")
131582025-04-30T06:43:15.656ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
131592025-04-30T06:43:15.661ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
131602025-04-30T06:43:15.662ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
131612025-04-30T06:43:15.662ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
131622025-04-30T06:43:15.663ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
131632025-04-30T06:43:15.666ZDEBGtest_snapshot_basic (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = [(d1ad44e5-18dd-4193-a70e-bf9042f0e0e1, 0)]
131642025-04-30T06:43:15.666ZTRACtest_snapshot_basic (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
131652025-04-30T06:43:15.666ZTRACtest_snapshot_basic (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: e9 de 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 ............._mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:56348 req_id = 85e55791-adbb-433f-a871-20ca5d47ddf4
131662025-04-30T06:43:15.666ZDEBGtest_snapshot_basic (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 59870,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:56348 req_id = 85e55791-adbb-433f-a871-20ca5d47ddf4
131672025-04-30T06:43:15.667ZDEBGtest_snapshot_basic (store): query key key = _mgs._tcp
131682025-04-30T06:43:15.667ZDEBGtest_snapshot_basic (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
131692025-04-30T06:43:15.667ZDEBGtest_snapshot_basic (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:56348 query = LowerQuery { name: LowerName(Name("_mgs._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 41502, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = 85e55791-adbb-433f-a871-20ca5d47ddf4
131702025-04-30T06:43:15.667ZDEBGtest_snapshot_basic (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 41502, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 4260, tv_nsec: 466709225 } })
131712025-04-30T06:43:15.667ZTRACtest_snapshot_basic (DnsResolver): lookup_service_targets: looking up SRV target
131722025-04-30T06:43:15.668ZTRACtest_snapshot_basic (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: 69 30 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e i0..........-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:61097 req_id = 57351591-8669-4181-89f6-3c3f504b2f99
131732025-04-30T06:43:15.668ZDEBGtest_snapshot_basic (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 26928,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:61097 req_id = 57351591-8669-4181-89f6-3c3f504b2f99
131742025-04-30T06:43:15.668ZDEBGtest_snapshot_basic (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
131752025-04-30T06:43:15.668ZDEBGtest_snapshot_basic (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:61097 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = 57351591-8669-4181-89f6-3c3f504b2f99
131762025-04-30T06:43:15.757ZTRACtest_snapshot_basic (DnsResolver): lookup_all_socket_v6 srv dns_name = _clickhouse-admin-keeper._tcp.control-plane.oxide.internal
131772025-04-30T06:43:15.758ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
131782025-04-30T06:43:15.758ZTRACtest_snapshot_basic (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
131792025-04-30T06:43:15.758ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
131802025-04-30T06:43:15.758ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {} }
131812025-04-30T06:43:15.759ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
131822025-04-30T06:43:15.759ZINFOtest_snapshot_basic (oximeter-agent): refreshing list of producers from Nexus collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 file = oximeter/collector/src/agent.rs:562
131832025-04-30T06:43:15.761ZDEBGtest_snapshot_basic (oximeter): client request body = None method = GET uri = http://127.0.0.1:61468/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
131842025-04-30T06:43:15.761ZTRACtest_snapshot_basic (dropshot_internal): incoming request local_addr = 127.0.0.1:61468 method = GET remote_addr = 127.0.0.1:64332 req_id = f0505bcc-7bad-49be-ba45-84d6e8e3179a uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
131852025-04-30T06:43:15.762ZTRACtest_snapshot_basic (dns): buffer buffer = Length: 87 (0x57) bytes\n0000: 64 6b 01 00 00 01 00 00 00 00 00 01 18 5f 63 6c dk..........._cl\n0010: 69 63 6b 68 6f 75 73 65 2d 61 64 6d 69 6e 2d 6b ickhouse-admin-k\n0020: 65 65 70 65 72 04 5f 74 63 70 0d 63 6f 6e 74 72 eeper._tcp.contr\n0030: 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 69 ol-plane.oxide.i\n0040: 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 04 nternal..!....).\n0050: d0 00 00 00 00 00 00 ....... kind = dns peer_addr = [::1]:53775 req_id = a7929818-0408-4846-bbda-c75b6f707bfd
131862025-04-30T06:43:15.762ZDEBGtest_snapshot_basic (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 25707,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 24,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 97,\n 100,\n 109,\n 105,\n 110,\n 45,\n 107,\n 101,\n 101,\n 112,\n 101,\n 114,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:53775 req_id = a7929818-0408-4846-bbda-c75b6f707bfd
131872025-04-30T06:43:15.762ZDEBGtest_snapshot_basic (store): query key key = _clickhouse-admin-keeper._tcp
131882025-04-30T06:43:15.762ZERROtest_snapshot_basic (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 25707, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } NXDOMAIN: no records found for name: "_clickhouse-admin-keeper._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:53775 req_id = a7929818-0408-4846-bbda-c75b6f707bfd
131892025-04-30T06:43:15.763ZTRACtest_snapshot_basic (dropshot_internal): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:61468 method = GET remote_addr = 127.0.0.1:64332 req_id = f0505bcc-7bad-49be-ba45-84d6e8e3179a resource = Database uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
131902025-04-30T06:43:15.763ZDEBGtest_snapshot_basic (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:61468 method = GET remote_addr = 127.0.0.1:64332 req_id = f0505bcc-7bad-49be-ba45-84d6e8e3179a roles = RoleSet { roles: {} } uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
131912025-04-30T06:43:15.764ZDEBGtest_snapshot_basic (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:61468 method = GET remote_addr = 127.0.0.1:64332 req_id = f0505bcc-7bad-49be-ba45-84d6e8e3179a resource = Database result = Ok(()) uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
131922025-04-30T06:43:15.765ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
131932025-04-30T06:43:15.771ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
131942025-04-30T06:43:15.771ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
131952025-04-30T06:43:15.771ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {} }
131962025-04-30T06:43:15.772ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
131972025-04-30T06:43:15.772ZDEBGtest_snapshot_basic (BackgroundTasks): begin collection background_task = inventory_collection
131982025-04-30T06:43:15.772ZDEBGtest_snapshot_basic (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:41502
131992025-04-30T06:43:15.772ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/ignition
132002025-04-30T06:43:15.776ZINFOtest_snapshot_basic (dropshot_internal): request completed latency_us = 14351 local_addr = 127.0.0.1:61468 method = GET remote_addr = 127.0.0.1:64332 req_id = f0505bcc-7bad-49be-ba45-84d6e8e3179a response_code = 200 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
132012025-04-30T06:43:15.777ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
132022025-04-30T06:43:15.777ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {} }
132032025-04-30T06:43:15.778ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
132042025-04-30T06:43:15.778ZDEBGtest_snapshot_basic (oximeter): client response result = Ok(Response { url: "http://127.0.0.1:61468/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending", status: 200, headers: {"content-type": "application/json", "x-request-id": "f0505bcc-7bad-49be-ba45-84d6e8e3179a", "content-length": "405", "date": "Wed, 30 Apr 2025 06:43:15 GMT"} })
132052025-04-30T06:43:15.779ZDEBGtest_snapshot_basic (oximeter): client request body = None method = GET uri = http://127.0.0.1:61468/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
132062025-04-30T06:43:15.779ZTRACtest_snapshot_basic (dropshot_internal): incoming request local_addr = 127.0.0.1:61468 method = GET remote_addr = 127.0.0.1:64332 req_id = e8cf51ea-d8bd-4fd4-bfb8-334f61b1a059 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
132072025-04-30T06:43:15.780ZTRACtest_snapshot_basic (dropshot_internal): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:61468 method = GET remote_addr = 127.0.0.1:64332 req_id = e8cf51ea-d8bd-4fd4-bfb8-334f61b1a059 resource = Database uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
132082025-04-30T06:43:15.780ZDEBGtest_snapshot_basic (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:61468 method = GET remote_addr = 127.0.0.1:64332 req_id = e8cf51ea-d8bd-4fd4-bfb8-334f61b1a059 roles = RoleSet { roles: {} } uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
132092025-04-30T06:43:15.780ZDEBGtest_snapshot_basic (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:61468 method = GET remote_addr = 127.0.0.1:64332 req_id = e8cf51ea-d8bd-4fd4-bfb8-334f61b1a059 resource = Database result = Ok(()) uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
132102025-04-30T06:43:15.781ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "3f60ebdb-aec5-4416-99f1-212f0d22053f", "content-length": "835", "date": "Wed, 30 Apr 2025 06:43:15 GMT"} })
132112025-04-30T06:43:15.782ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/0
132122025-04-30T06:43:15.784ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "5cb9a0fb-27f1-4063-8fe5-ce01a7a4c54d", "content-length": "734", "date": "Wed, 30 Apr 2025 06:43:15 GMT"} })
132132025-04-30T06:43:15.784ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/0/component/sp/caboose?firmware_slot=0
132142025-04-30T06:43:15.785ZINFOtest_snapshot_basic (dropshot_internal): request completed latency_us = 5882 local_addr = 127.0.0.1:61468 method = GET remote_addr = 127.0.0.1:64332 req_id = e8cf51ea-d8bd-4fd4-bfb8-334f61b1a059 response_code = 200 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
132152025-04-30T06:43:15.786ZDEBGtest_snapshot_basic (oximeter): client response result = Ok(Response { url: "http://127.0.0.1:61468/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D", status: 200, headers: {"content-type": "application/json", "x-request-id": "e8cf51ea-d8bd-4fd4-bfb8-334f61b1a059", "content-length": "29", "date": "Wed, 30 Apr 2025 06:43:15 GMT"} })
132162025-04-30T06:43:15.786ZTRACtest_snapshot_basic (oximeter-agent): ignoring request to update existing metric producer, since the endpoint information is the same as the existing collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = 354a52d4-3078-4706-81ce-12c32f6bbeaa
132172025-04-30T06:43:15.786ZTRACtest_snapshot_basic (oximeter-agent): ignoring request to update existing metric producer, since the endpoint information is the same as the existing collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
132182025-04-30T06:43:15.787ZINFOtest_snapshot_basic (oximeter-agent): refreshed list of producers from Nexus collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 file = oximeter/collector/src/agent.rs:637 n_current_tasks = 2 n_pruned_tasks = 0
132192025-04-30T06:43:15.789ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "01e02b5f-1817-4bdb-b100-27b17ae1898a", "content-length": "111", "date": "Wed, 30 Apr 2025 06:43:15 GMT"} })
132202025-04-30T06:43:15.789ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/0/component/sp/caboose?firmware_slot=1
132212025-04-30T06:43:15.791ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "fd63d64b-6699-4128-9cb0-c8114e190bfc", "content-length": "111", "date": "Wed, 30 Apr 2025 06:43:15 GMT"} })
132222025-04-30T06:43:15.791ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/0/component/rot/caboose?firmware_slot=0
132232025-04-30T06:43:15.796ZDEBGtest_snapshot_basic (collection-task): sent timer-based collection request to the collection task collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
132242025-04-30T06:43:15.796ZDEBGtest_snapshot_basic (collection-task): collecting from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
132252025-04-30T06:43:15.801ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "456c2b8e-516a-486a-8435-4177c09ae6cf", "content-length": "170", "date": "Wed, 30 Apr 2025 06:43:15 GMT"} })
132262025-04-30T06:43:15.801ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/0/component/rot/caboose?firmware_slot=1
132272025-04-30T06:43:15.801ZTRACtest_snapshot_basic (collection-task): sent collection request to producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
132282025-04-30T06:43:15.805ZDEBGtest_snapshot_basic (collection-task): collected results from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 n_results = 1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
132292025-04-30T06:43:15.805ZTRACtest_snapshot_basic (collection-task): forwarded results to main collection loop collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
132302025-04-30T06:43:15.805ZTRACtest_snapshot_basic (collection-task): top of inner collection loop, waiting for next request collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
132312025-04-30T06:43:15.813ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "cd298951-eecd-420f-96e0-f16ad8587d7a", "content-length": "170", "date": "Wed, 30 Apr 2025 06:43:15 GMT"} })
132322025-04-30T06:43:15.813ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/0/component/stage0/caboose?firmware_slot=0
132332025-04-30T06:43:15.820ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "2744f33a-dbc8-4875-9aad-8f0e46df8d34", "content-length": "179", "date": "Wed, 30 Apr 2025 06:43:15 GMT"} })
132342025-04-30T06:43:15.820ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/0/component/stage0/caboose?firmware_slot=1
132352025-04-30T06:43:15.821ZDEBGtest_snapshot_basic: attempt to fetch latest inventory collection result = Ok("found")
132362025-04-30T06:43:15.827ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "84319187-6b87-4f26-9e61-908a2d2a72d9", "content-length": "179", "date": "Wed, 30 Apr 2025 06:43:15 GMT"} })
132372025-04-30T06:43:15.827ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/0/component/rot/cmpa
132382025-04-30T06:43:15.829ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "b2b7fa0e-cc50-48c1-85ab-77cfc5a1964d", "content-length": "702", "date": "Wed, 30 Apr 2025 06:43:15 GMT"} })
132392025-04-30T06:43:15.829ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/0/component/rot/cfpa
132402025-04-30T06:43:15.831ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "96d8734b-aed6-46df-97ce-8df4a8741a6e", "content-length": "727", "date": "Wed, 30 Apr 2025 06:43:15 GMT"} })
132412025-04-30T06:43:15.832ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/0/component/rot/cfpa
132422025-04-30T06:43:15.833ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "4e08b50c-fc49-4626-95e2-b6ba74080d3d", "content-length": "729", "date": "Wed, 30 Apr 2025 06:43:15 GMT"} })
132432025-04-30T06:43:15.833ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/0/component/rot/cfpa
132442025-04-30T06:43:15.834ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "7d8318d7-401b-4ad0-8947-7b9bc0206c7e", "content-length": "728", "date": "Wed, 30 Apr 2025 06:43:15 GMT"} })
132452025-04-30T06:43:15.834ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/1
132462025-04-30T06:43:15.835ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "81423119-be8b-4b9e-b641-102f01d57df4", "content-length": "734", "date": "Wed, 30 Apr 2025 06:43:15 GMT"} })
132472025-04-30T06:43:15.835ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/1/component/sp/caboose?firmware_slot=0
132482025-04-30T06:43:15.837ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "f438ce82-107a-456d-81de-d62f13e7f0aa", "content-length": "111", "date": "Wed, 30 Apr 2025 06:43:15 GMT"} })
132492025-04-30T06:43:15.837ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/1/component/sp/caboose?firmware_slot=1
132502025-04-30T06:43:15.840ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "ad8fec8f-7806-479a-b0f1-c3b760e9013c", "content-length": "111", "date": "Wed, 30 Apr 2025 06:43:15 GMT"} })
132512025-04-30T06:43:15.840ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/1/component/rot/caboose?firmware_slot=0
132522025-04-30T06:43:15.842ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "2767c054-1515-42da-a9d9-be56d73bd182", "content-length": "170", "date": "Wed, 30 Apr 2025 06:43:15 GMT"} })
132532025-04-30T06:43:15.843ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/1/component/rot/caboose?firmware_slot=1
132542025-04-30T06:43:15.845ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "af2cca56-2537-446b-a10e-7ca0c4752437", "content-length": "170", "date": "Wed, 30 Apr 2025 06:43:15 GMT"} })
132552025-04-30T06:43:15.845ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/1/component/stage0/caboose?firmware_slot=0
132562025-04-30T06:43:15.857ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "70a0b42b-4ac3-4eac-a53a-d6513ec45edd", "content-length": "179", "date": "Wed, 30 Apr 2025 06:43:15 GMT"} })
132572025-04-30T06:43:15.857ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/1/component/stage0/caboose?firmware_slot=1
132582025-04-30T06:43:15.860ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "78b3a43f-6422-4522-9e25-cd8cbe1135d5", "content-length": "179", "date": "Wed, 30 Apr 2025 06:43:15 GMT"} })
132592025-04-30T06:43:15.860ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/1/component/rot/cmpa
132602025-04-30T06:43:15.861ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "0559230f-874a-47e0-93ac-cae1bf1560ea", "content-length": "702", "date": "Wed, 30 Apr 2025 06:43:15 GMT"} })
132612025-04-30T06:43:15.861ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/1/component/rot/cfpa
132622025-04-30T06:43:15.863ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "a50db47d-6401-4c8f-8436-3f43e1449332", "content-length": "727", "date": "Wed, 30 Apr 2025 06:43:15 GMT"} })
132632025-04-30T06:43:15.863ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/1/component/rot/cfpa
132642025-04-30T06:43:15.864ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "50d90f85-c094-4020-88c6-bf56262d4a59", "content-length": "729", "date": "Wed, 30 Apr 2025 06:43:15 GMT"} })
132652025-04-30T06:43:15.864ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/1/component/rot/cfpa
132662025-04-30T06:43:15.866ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "2b357a62-18a2-4331-a18a-ba1716038526", "content-length": "728", "date": "Wed, 30 Apr 2025 06:43:15 GMT"} })
132672025-04-30T06:43:15.866ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/0
132682025-04-30T06:43:15.867ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "b8b015d0-3644-4686-aada-d8cb0bbfcf1a", "content-length": "723", "date": "Wed, 30 Apr 2025 06:43:15 GMT"} })
132692025-04-30T06:43:15.868ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/0/component/sp/caboose?firmware_slot=0
132702025-04-30T06:43:15.871ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "52ca4a2d-cf38-4eff-bd11-7898ba1159c0", "content-length": "109", "date": "Wed, 30 Apr 2025 06:43:15 GMT"} })
132712025-04-30T06:43:15.871ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/0/component/sp/caboose?firmware_slot=1
132722025-04-30T06:43:15.872ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
132732025-04-30T06:43:15.872ZTRACtest_snapshot_basic (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
132742025-04-30T06:43:15.872ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
132752025-04-30T06:43:15.872ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {} }
132762025-04-30T06:43:15.873ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
132772025-04-30T06:43:15.875ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
132782025-04-30T06:43:15.881ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
132792025-04-30T06:43:15.881ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
132802025-04-30T06:43:15.881ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {} }
132812025-04-30T06:43:15.881ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
132822025-04-30T06:43:15.881ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "add611e7-5435-446c-8b9d-9fccd38a1ee8", "content-length": "109", "date": "Wed, 30 Apr 2025 06:43:15 GMT"} })
132832025-04-30T06:43:15.881ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/0/component/rot/caboose?firmware_slot=0
132842025-04-30T06:43:15.885ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
132852025-04-30T06:43:15.885ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {} }
132862025-04-30T06:43:15.885ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
132872025-04-30T06:43:15.888ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "ad16e703-b7b7-496f-8e41-c16beb72b1a9", "content-length": "169", "date": "Wed, 30 Apr 2025 06:43:15 GMT"} })
132882025-04-30T06:43:15.888ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/0/component/rot/caboose?firmware_slot=1
132892025-04-30T06:43:15.891ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "3e4c78cd-9d95-4cb7-a980-d777efdd7cd9", "content-length": "169", "date": "Wed, 30 Apr 2025 06:43:15 GMT"} })
132902025-04-30T06:43:15.891ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/0/component/stage0/caboose?firmware_slot=0
132912025-04-30T06:43:15.893ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "aba5e094-626c-4a4a-82c1-33134895ea9a", "content-length": "178", "date": "Wed, 30 Apr 2025 06:43:15 GMT"} })
132922025-04-30T06:43:15.893ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/0/component/stage0/caboose?firmware_slot=1
132932025-04-30T06:43:15.895ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "85b3e8c9-655c-4b5e-8671-e636574b347c", "content-length": "178", "date": "Wed, 30 Apr 2025 06:43:15 GMT"} })
132942025-04-30T06:43:15.895ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/0/component/rot/cmpa
132952025-04-30T06:43:15.897ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "bd8b5ecc-0235-451d-be30-37dd15fbf2d7", "content-length": "702", "date": "Wed, 30 Apr 2025 06:43:15 GMT"} })
132962025-04-30T06:43:15.897ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/0/component/rot/cfpa
132972025-04-30T06:43:15.899ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "d98a05a7-8531-4f8b-8d8b-effa8b14b538", "content-length": "727", "date": "Wed, 30 Apr 2025 06:43:15 GMT"} })
132982025-04-30T06:43:15.899ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/0/component/rot/cfpa
132992025-04-30T06:43:15.900ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "f980aa0b-d807-45fb-b110-5a4af37017bb", "content-length": "729", "date": "Wed, 30 Apr 2025 06:43:15 GMT"} })
133002025-04-30T06:43:15.900ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/0/component/rot/cfpa
133012025-04-30T06:43:15.902ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "1c1632d3-1003-4669-943a-f0e2d3edfbd0", "content-length": "728", "date": "Wed, 30 Apr 2025 06:43:15 GMT"} })
133022025-04-30T06:43:15.903ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/1
133032025-04-30T06:43:15.905ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "2d5d7658-33f0-42e8-bd94-a82481dc5f9b", "content-length": "723", "date": "Wed, 30 Apr 2025 06:43:15 GMT"} })
133042025-04-30T06:43:15.905ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/1/component/sp/caboose?firmware_slot=0
133052025-04-30T06:43:15.908ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "7422bfcc-3f33-43ba-91ea-b8a58d6b201c", "content-length": "109", "date": "Wed, 30 Apr 2025 06:43:15 GMT"} })
133062025-04-30T06:43:15.908ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/1/component/sp/caboose?firmware_slot=1
133072025-04-30T06:43:15.911ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "5530551d-265b-4c9a-93b6-f7808357fdac", "content-length": "109", "date": "Wed, 30 Apr 2025 06:43:15 GMT"} })
133082025-04-30T06:43:15.911ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/1/component/rot/caboose?firmware_slot=0
133092025-04-30T06:43:15.917ZDEBGtest_snapshot_basic: attempt to fetch latest inventory collection result = Ok("found")
133102025-04-30T06:43:15.920ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "6387fb86-acaa-427d-a983-1de885751222", "content-length": "169", "date": "Wed, 30 Apr 2025 06:43:15 GMT"} })
133112025-04-30T06:43:15.920ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/1/component/rot/caboose?firmware_slot=1
133122025-04-30T06:43:15.922ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "d3f346aa-3a4f-4754-ad24-6ce53894165f", "content-length": "169", "date": "Wed, 30 Apr 2025 06:43:15 GMT"} })
133132025-04-30T06:43:15.922ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/1/component/stage0/caboose?firmware_slot=0
133142025-04-30T06:43:15.924ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "a3b3eccb-8b32-4ec8-9199-07ca6dd46bb2", "content-length": "178", "date": "Wed, 30 Apr 2025 06:43:15 GMT"} })
133152025-04-30T06:43:15.924ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/1/component/stage0/caboose?firmware_slot=1
133162025-04-30T06:43:15.926ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "31834d3e-b357-435d-a742-de1c38e4fc9d", "content-length": "178", "date": "Wed, 30 Apr 2025 06:43:15 GMT"} })
133172025-04-30T06:43:15.926ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/1/component/rot/cmpa
133182025-04-30T06:43:15.928ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "cb02b694-d094-464d-9242-7f074167af01", "content-length": "702", "date": "Wed, 30 Apr 2025 06:43:15 GMT"} })
133192025-04-30T06:43:15.928ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/1/component/rot/cfpa
133202025-04-30T06:43:15.931ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "d39bfb25-fb5f-4a33-a592-baa5c2d553fa", "content-length": "727", "date": "Wed, 30 Apr 2025 06:43:15 GMT"} })
133212025-04-30T06:43:15.932ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/1/component/rot/cfpa
133222025-04-30T06:43:15.933ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "b9bd150d-d0a8-48ba-9ec6-7956d6d55f86", "content-length": "729", "date": "Wed, 30 Apr 2025 06:43:15 GMT"} })
133232025-04-30T06:43:15.934ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/1/component/rot/cfpa
133242025-04-30T06:43:15.935ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "22fa584a-b634-4f4d-949f-a2d15239e187", "content-length": "728", "date": "Wed, 30 Apr 2025 06:43:15 GMT"} })
133252025-04-30T06:43:15.935ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
133262025-04-30T06:43:15.935ZTRACtest_snapshot_basic (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
133272025-04-30T06:43:15.935ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
133282025-04-30T06:43:15.935ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
133292025-04-30T06:43:15.935ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
133302025-04-30T06:43:15.939ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
133312025-04-30T06:43:15.944ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
133322025-04-30T06:43:15.944ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
133332025-04-30T06:43:15.944ZTRACtest_snapshot_basic (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
133342025-04-30T06:43:15.944ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
133352025-04-30T06:43:15.944ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
133362025-04-30T06:43:15.945ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
133372025-04-30T06:43:15.956ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
133382025-04-30T06:43:15.963ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
133392025-04-30T06:43:15.963ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
133402025-04-30T06:43:15.964ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
133412025-04-30T06:43:15.966ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
133422025-04-30T06:43:15.969ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
133432025-04-30T06:43:15.969ZTRACtest_snapshot_basic (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
133442025-04-30T06:43:15.969ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
133452025-04-30T06:43:15.969ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {} }
133462025-04-30T06:43:15.970ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
133472025-04-30T06:43:15.974ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
133482025-04-30T06:43:15.980ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
133492025-04-30T06:43:15.980ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
133502025-04-30T06:43:15.980ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {} }
133512025-04-30T06:43:15.981ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
133522025-04-30T06:43:16.054ZDEBGtest_snapshot_basic (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:41923
133532025-04-30T06:43:16.054ZDEBGtest_snapshot_basic (BackgroundTasks): client request SledAgent = http://[::1]:41923 background_task = inventory_collection body = None method = GET uri = http://[::1]:41923/inventory
133542025-04-30T06:43:16.056ZINFOtest_snapshot_basic (dropshot): accepted connection local_addr = [::1]:41923 remote_addr = [::1]:50013 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
133552025-04-30T06:43:16.056ZDEBGtest_snapshot_basic (results-sink-cluster): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
133562025-04-30T06:43:16.056ZDEBGtest_snapshot_basic (results-sink): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
133572025-04-30T06:43:16.056ZDEBGtest_snapshot_basic (clickhouse-client): unrolling 1 total samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 6ad06c6b-9a6e-4ed1-b3f4-5d2e9da3f8a7
133582025-04-30T06:43:16.057ZTRACtest_snapshot_basic (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 6ad06c6b-9a6e-4ed1-b3f4-5d2e9da3f8a7 n_columns = 4 n_rows = 2 sql = INSERT INTO oximeter.fields_string FORMAT Native
133592025-04-30T06:43:16.057ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
133602025-04-30T06:43:16.057ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {} }
133612025-04-30T06:43:16.059ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
133622025-04-30T06:43:16.059ZTRACtest_snapshot_basic (dropshot): incoming request local_addr = [::1]:41923 method = GET remote_addr = [::1]:50013 req_id = 3d5b45c3-5c1f-4900-b0b0-7aae7e0e5106 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
133632025-04-30T06:43:16.060ZINFOtest_snapshot_basic (dropshot): request completed latency_us = 696 local_addr = [::1]:41923 method = GET remote_addr = [::1]:50013 req_id = 3d5b45c3-5c1f-4900-b0b0-7aae7e0e5106 response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
133642025-04-30T06:43:16.060ZDEBGtest_snapshot_basic (BackgroundTasks): client response SledAgent = http://[::1]:41923 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:41923/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "3d5b45c3-5c1f-4900-b0b0-7aae7e0e5106", "content-length": "3141", "date": "Wed, 30 Apr 2025 06:43:15 GMT"} })
133652025-04-30T06:43:16.064ZDEBGtest_snapshot_basic (BackgroundTasks): begin collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
133662025-04-30T06:43:16.064ZDEBGtest_snapshot_basic (BackgroundTasks): end collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
133672025-04-30T06:43:16.064ZDEBGtest_snapshot_basic (BackgroundTasks): finished collection background_task = inventory_collection
133682025-04-30T06:43:16.066ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
133692025-04-30T06:43:16.066ZTRACtest_snapshot_basic (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
133702025-04-30T06:43:16.066ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
133712025-04-30T06:43:16.066ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
133722025-04-30T06:43:16.067ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
133732025-04-30T06:43:16.068ZDEBGtest_snapshot_basic (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 6ad06c6b-9a6e-4ed1-b3f4-5d2e9da3f8a7 n_rows = 2 table_name = fields_string
133742025-04-30T06:43:16.068ZTRACtest_snapshot_basic (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 6ad06c6b-9a6e-4ed1-b3f4-5d2e9da3f8a7 n_columns = 4 n_rows = 1 sql = INSERT INTO oximeter.measurements_i64 FORMAT Native
133752025-04-30T06:43:16.072ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
133762025-04-30T06:43:16.079ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
133772025-04-30T06:43:16.079ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
133782025-04-30T06:43:16.079ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
133792025-04-30T06:43:16.080ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
133802025-04-30T06:43:16.081ZDEBGtest_snapshot_basic (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 6ad06c6b-9a6e-4ed1-b3f4-5d2e9da3f8a7 n_rows = 1 table_name = measurements_i64
133812025-04-30T06:43:16.081ZTRACtest_snapshot_basic (results-sink): successfully inserted samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
133822025-04-30T06:43:16.120ZDEBGtest_snapshot_basic: attempt to fetch latest inventory collection result = Ok("found")
133832025-04-30T06:43:16.133ZWARNtest_snapshot_basic (datastore): Non-retryable transaction failure err = DatabaseError(UniqueViolation, "duplicate key value violates unique constraint \\"inv_caboose_pkey\\"") transaction = inventory_insert_collection
133842025-04-30T06:43:16.134ZWARNtest_snapshot_basic (BackgroundTasks): inventory collection failed background_task = inventory_collection error = failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint "inv_caboose_pkey"
133852025-04-30T06:43:16.134ZDEBGtest_snapshot_basic (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 485.299467ms iteration = 48 status = {"error":"failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint \\"inv_caboose_pkey\\""}
133862025-04-30T06:43:16.134ZDEBGtest_snapshot_basic (BackgroundTasks): activating background_task = inventory_collection iteration = 49 reason = Signaled
133872025-04-30T06:43:16.134ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
133882025-04-30T06:43:16.134ZTRACtest_snapshot_basic (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
133892025-04-30T06:43:16.134ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
133902025-04-30T06:43:16.134ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
133912025-04-30T06:43:16.135ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
133922025-04-30T06:43:16.138ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
133932025-04-30T06:43:16.145ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
133942025-04-30T06:43:16.145ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
133952025-04-30T06:43:16.145ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
133962025-04-30T06:43:16.145ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
133972025-04-30T06:43:16.150ZDEBGtest_snapshot_basic (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = [(d1ad44e5-18dd-4193-a70e-bf9042f0e0e1, 0)]
133982025-04-30T06:43:16.150ZTRACtest_snapshot_basic (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
133992025-04-30T06:43:16.154ZTRACtest_snapshot_basic (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: 1e 36 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 .6..........._mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:52464 req_id = 5b6229dc-677d-4a06-80e7-b7a3ea77d94f
134002025-04-30T06:43:16.158ZDEBGtest_snapshot_basic (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 7734,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:52464 req_id = 5b6229dc-677d-4a06-80e7-b7a3ea77d94f
134012025-04-30T06:43:16.159ZDEBGtest_snapshot_basic (store): query key key = _mgs._tcp
134022025-04-30T06:43:16.163ZDEBGtest_snapshot_basic (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
134032025-04-30T06:43:16.165ZDEBGtest_snapshot_basic (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:52464 query = LowerQuery { name: LowerName(Name("_mgs._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 41502, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = 5b6229dc-677d-4a06-80e7-b7a3ea77d94f
134042025-04-30T06:43:16.166ZDEBGtest_snapshot_basic (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 41502, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 4260, tv_nsec: 965326066 } })
134052025-04-30T06:43:16.166ZTRACtest_snapshot_basic (DnsResolver): lookup_service_targets: looking up SRV target
134062025-04-30T06:43:16.167ZWARNtest_snapshot_basic (results-sink-cluster): failed to insert some results into metric DB collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 error = Failed to check out connection to database: No backends found for this service file = oximeter/collector/src/results_sink.rs:92
134072025-04-30T06:43:16.169ZTRACtest_snapshot_basic (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: c9 e5 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e ............-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:62086 req_id = 32d3ba0b-ca6b-419d-a98f-6f13fac67d93
134082025-04-30T06:43:16.169ZDEBGtest_snapshot_basic (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 51685,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:62086 req_id = 32d3ba0b-ca6b-419d-a98f-6f13fac67d93
134092025-04-30T06:43:16.169ZDEBGtest_snapshot_basic (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
134102025-04-30T06:43:16.169ZDEBGtest_snapshot_basic (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:62086 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = 32d3ba0b-ca6b-419d-a98f-6f13fac67d93
134112025-04-30T06:43:16.242ZTRACtest_snapshot_basic (DnsResolver): lookup_all_socket_v6 srv dns_name = _clickhouse-admin-keeper._tcp.control-plane.oxide.internal
134122025-04-30T06:43:16.245ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
134132025-04-30T06:43:16.245ZTRACtest_snapshot_basic (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
134142025-04-30T06:43:16.245ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
134152025-04-30T06:43:16.245ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {} }
134162025-04-30T06:43:16.246ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
134172025-04-30T06:43:16.248ZTRACtest_snapshot_basic (dns): buffer buffer = Length: 87 (0x57) bytes\n0000: 1e 9a 01 00 00 01 00 00 00 00 00 01 18 5f 63 6c ............._cl\n0010: 69 63 6b 68 6f 75 73 65 2d 61 64 6d 69 6e 2d 6b ickhouse-admin-k\n0020: 65 65 70 65 72 04 5f 74 63 70 0d 63 6f 6e 74 72 eeper._tcp.contr\n0030: 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 69 ol-plane.oxide.i\n0040: 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 04 nternal..!....).\n0050: d0 00 00 00 00 00 00 ....... kind = dns peer_addr = [::1]:52931 req_id = 2f5b3dc0-cfb2-4ab3-8573-aeee88dfdd49
134182025-04-30T06:43:16.248ZDEBGtest_snapshot_basic (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 7834,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 24,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 97,\n 100,\n 109,\n 105,\n 110,\n 45,\n 107,\n 101,\n 101,\n 112,\n 101,\n 114,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:52931 req_id = 2f5b3dc0-cfb2-4ab3-8573-aeee88dfdd49
134192025-04-30T06:43:16.248ZDEBGtest_snapshot_basic (store): query key key = _clickhouse-admin-keeper._tcp
134202025-04-30T06:43:16.248ZERROtest_snapshot_basic (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 7834, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } NXDOMAIN: no records found for name: "_clickhouse-admin-keeper._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:52931 req_id = 2f5b3dc0-cfb2-4ab3-8573-aeee88dfdd49
134212025-04-30T06:43:16.250ZDEBGtest_snapshot_basic (BackgroundTasks): begin collection background_task = inventory_collection
134222025-04-30T06:43:16.250ZDEBGtest_snapshot_basic (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:41502
134232025-04-30T06:43:16.252ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/ignition
134242025-04-30T06:43:16.252ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "3c15a6ca-7aa1-439d-8068-e3d8b69ee274", "content-length": "835", "date": "Wed, 30 Apr 2025 06:43:16 GMT"} })
134252025-04-30T06:43:16.252ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/0
134262025-04-30T06:43:16.252ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
134272025-04-30T06:43:16.258ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
134282025-04-30T06:43:16.258ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
134292025-04-30T06:43:16.258ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {} }
134302025-04-30T06:43:16.258ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
134312025-04-30T06:43:16.261ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "f082cc03-cf67-4ce8-a7b1-872e348e08d2", "content-length": "734", "date": "Wed, 30 Apr 2025 06:43:16 GMT"} })
134322025-04-30T06:43:16.261ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/0/component/sp/caboose?firmware_slot=0
134332025-04-30T06:43:16.261ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
134342025-04-30T06:43:16.261ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {} }
134352025-04-30T06:43:16.263ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
134362025-04-30T06:43:16.266ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "76d911d1-987a-43b0-ae7b-6b4e0d27713f", "content-length": "111", "date": "Wed, 30 Apr 2025 06:43:16 GMT"} })
134372025-04-30T06:43:16.266ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/0/component/sp/caboose?firmware_slot=1
134382025-04-30T06:43:16.268ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "37b3605a-03ee-4c98-912f-c9dd60417254", "content-length": "111", "date": "Wed, 30 Apr 2025 06:43:16 GMT"} })
134392025-04-30T06:43:16.268ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/0/component/rot/caboose?firmware_slot=0
134402025-04-30T06:43:16.270ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "92784f05-cf94-45d3-b3e2-9abca737ee1d", "content-length": "170", "date": "Wed, 30 Apr 2025 06:43:16 GMT"} })
134412025-04-30T06:43:16.270ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/0/component/rot/caboose?firmware_slot=1
134422025-04-30T06:43:16.272ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "a64ecc76-799d-48fb-9bd6-644ace9ebd92", "content-length": "170", "date": "Wed, 30 Apr 2025 06:43:16 GMT"} })
134432025-04-30T06:43:16.273ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/0/component/stage0/caboose?firmware_slot=0
134442025-04-30T06:43:16.275ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "fe081a96-f420-4a99-bd60-660516361551", "content-length": "179", "date": "Wed, 30 Apr 2025 06:43:16 GMT"} })
134452025-04-30T06:43:16.275ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/0/component/stage0/caboose?firmware_slot=1
134462025-04-30T06:43:16.277ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "cbda7f76-5ec1-4507-ad43-a0340eb1976a", "content-length": "179", "date": "Wed, 30 Apr 2025 06:43:16 GMT"} })
134472025-04-30T06:43:16.277ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/0/component/rot/cmpa
134482025-04-30T06:43:16.278ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "1d703553-ce92-4f13-8d37-5621e23e2ca3", "content-length": "702", "date": "Wed, 30 Apr 2025 06:43:16 GMT"} })
134492025-04-30T06:43:16.278ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/0/component/rot/cfpa
134502025-04-30T06:43:16.280ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "0611206a-e5f1-4dbd-9850-50cad85df3ac", "content-length": "727", "date": "Wed, 30 Apr 2025 06:43:16 GMT"} })
134512025-04-30T06:43:16.280ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/0/component/rot/cfpa
134522025-04-30T06:43:16.282ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "7dd09f28-7842-4b9e-8923-72007f327434", "content-length": "729", "date": "Wed, 30 Apr 2025 06:43:16 GMT"} })
134532025-04-30T06:43:16.283ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/0/component/rot/cfpa
134542025-04-30T06:43:16.284ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "a66d99c7-bde0-43ee-9062-39ff36b2a134", "content-length": "728", "date": "Wed, 30 Apr 2025 06:43:16 GMT"} })
134552025-04-30T06:43:16.284ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/1
134562025-04-30T06:43:16.287ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "759677a8-db56-44c9-8230-3307e7a2a46f", "content-length": "734", "date": "Wed, 30 Apr 2025 06:43:16 GMT"} })
134572025-04-30T06:43:16.287ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/1/component/sp/caboose?firmware_slot=0
134582025-04-30T06:43:16.293ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "919d4b4e-e5f3-4d77-ac7e-12f269551657", "content-length": "111", "date": "Wed, 30 Apr 2025 06:43:16 GMT"} })
134592025-04-30T06:43:16.293ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/1/component/sp/caboose?firmware_slot=1
134602025-04-30T06:43:16.298ZDEBGtest_snapshot_basic: attempt to fetch latest inventory collection result = Ok("found")
134612025-04-30T06:43:16.303ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "36fedfc4-d914-41bf-9820-17db46ee97cc", "content-length": "111", "date": "Wed, 30 Apr 2025 06:43:16 GMT"} })
134622025-04-30T06:43:16.303ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/1/component/rot/caboose?firmware_slot=0
134632025-04-30T06:43:16.306ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "b9fc5971-3eb9-40e5-ba80-d5d0d46b365c", "content-length": "170", "date": "Wed, 30 Apr 2025 06:43:16 GMT"} })
134642025-04-30T06:43:16.306ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/1/component/rot/caboose?firmware_slot=1
134652025-04-30T06:43:16.309ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "cc93ab27-3608-4914-b2cf-e284902917e2", "content-length": "170", "date": "Wed, 30 Apr 2025 06:43:16 GMT"} })
134662025-04-30T06:43:16.309ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/1/component/stage0/caboose?firmware_slot=0
134672025-04-30T06:43:16.311ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "c414df7c-3c13-4f5c-8fdb-8ab8af59fa8a", "content-length": "179", "date": "Wed, 30 Apr 2025 06:43:16 GMT"} })
134682025-04-30T06:43:16.319ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/1/component/stage0/caboose?firmware_slot=1
134692025-04-30T06:43:16.319ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "7bd5f32d-d5b1-450d-b7ba-99a6f3d295a6", "content-length": "179", "date": "Wed, 30 Apr 2025 06:43:16 GMT"} })
134702025-04-30T06:43:16.319ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/1/component/rot/cmpa
134712025-04-30T06:43:16.319ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "3f1e581a-c4c2-4c47-9e7e-8ae2ec8fb95f", "content-length": "702", "date": "Wed, 30 Apr 2025 06:43:16 GMT"} })
134722025-04-30T06:43:16.319ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/1/component/rot/cfpa
134732025-04-30T06:43:16.319ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "767144cb-afb9-4fbf-a18b-9102328002f3", "content-length": "727", "date": "Wed, 30 Apr 2025 06:43:16 GMT"} })
134742025-04-30T06:43:16.319ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/1/component/rot/cfpa
134752025-04-30T06:43:16.321ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "779765ac-78f5-417b-aeca-0c8cad1e5336", "content-length": "729", "date": "Wed, 30 Apr 2025 06:43:16 GMT"} })
134762025-04-30T06:43:16.321ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/1/component/rot/cfpa
134772025-04-30T06:43:16.322ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "003ea994-d7cc-4b0e-a394-37b8b78a6f82", "content-length": "728", "date": "Wed, 30 Apr 2025 06:43:16 GMT"} })
134782025-04-30T06:43:16.322ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/0
134792025-04-30T06:43:16.323ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "5b9b2cde-9b03-469e-a0b7-4e7921bcea68", "content-length": "723", "date": "Wed, 30 Apr 2025 06:43:16 GMT"} })
134802025-04-30T06:43:16.323ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/0/component/sp/caboose?firmware_slot=0
134812025-04-30T06:43:16.327ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "32b72465-3393-4f41-9902-c0e0ff2fd9c9", "content-length": "109", "date": "Wed, 30 Apr 2025 06:43:16 GMT"} })
134822025-04-30T06:43:16.327ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/0/component/sp/caboose?firmware_slot=1
134832025-04-30T06:43:16.329ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "34e23cb0-a277-463d-81ee-57713f514354", "content-length": "109", "date": "Wed, 30 Apr 2025 06:43:16 GMT"} })
134842025-04-30T06:43:16.329ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/0/component/rot/caboose?firmware_slot=0
134852025-04-30T06:43:16.331ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "26fe072e-1300-49cb-b384-af602f66da8c", "content-length": "169", "date": "Wed, 30 Apr 2025 06:43:16 GMT"} })
134862025-04-30T06:43:16.331ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/0/component/rot/caboose?firmware_slot=1
134872025-04-30T06:43:16.333ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "0a700f5f-c9a0-4179-b633-f7b1272d72cd", "content-length": "169", "date": "Wed, 30 Apr 2025 06:43:16 GMT"} })
134882025-04-30T06:43:16.333ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/0/component/stage0/caboose?firmware_slot=0
134892025-04-30T06:43:16.335ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "89ea83e6-3175-4f07-8a80-f5f78701cf6a", "content-length": "178", "date": "Wed, 30 Apr 2025 06:43:16 GMT"} })
134902025-04-30T06:43:16.335ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/0/component/stage0/caboose?firmware_slot=1
134912025-04-30T06:43:16.340ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "1ded2176-1902-4611-9465-1898a9a3d3d9", "content-length": "178", "date": "Wed, 30 Apr 2025 06:43:16 GMT"} })
134922025-04-30T06:43:16.340ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/0/component/rot/cmpa
134932025-04-30T06:43:16.343ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "4972cb03-a5f6-4ed4-bcda-f60354796473", "content-length": "702", "date": "Wed, 30 Apr 2025 06:43:16 GMT"} })
134942025-04-30T06:43:16.344ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/0/component/rot/cfpa
134952025-04-30T06:43:16.345ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "bb005146-f213-4db2-9169-8ccb216416f8", "content-length": "727", "date": "Wed, 30 Apr 2025 06:43:16 GMT"} })
134962025-04-30T06:43:16.345ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/0/component/rot/cfpa
134972025-04-30T06:43:16.346ZINFOtest_snapshot_basic (dropshot): accepted connection kind = pantry local_addr = [::1]:62733 remote_addr = [::1]:44138 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
134982025-04-30T06:43:16.347ZTRACtest_snapshot_basic (dropshot): incoming request kind = pantry local_addr = [::1]:62733 method = GET remote_addr = [::1]:44138 req_id = 9aed8477-d05f-41d1-b381-a4a5b69ad880 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /crucible/pantry/0
134992025-04-30T06:43:16.347ZINFOtest_snapshot_basic (dropshot): request completed kind = pantry latency_us = 335 local_addr = [::1]:62733 method = GET remote_addr = [::1]:44138 req_id = 9aed8477-d05f-41d1-b381-a4a5b69ad880 response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /crucible/pantry/0
135002025-04-30T06:43:16.348ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "d4e6957a-73df-46b7-8126-7c17e32dd57d", "content-length": "729", "date": "Wed, 30 Apr 2025 06:43:16 GMT"} })
135012025-04-30T06:43:16.348ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/0/component/rot/cfpa
135022025-04-30T06:43:16.349ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "908c5321-2e7e-4b89-be8f-da7f426ec907", "content-length": "728", "date": "Wed, 30 Apr 2025 06:43:16 GMT"} })
135032025-04-30T06:43:16.349ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/1
135042025-04-30T06:43:16.350ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
135052025-04-30T06:43:16.351ZTRACtest_snapshot_basic (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
135062025-04-30T06:43:16.351ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
135072025-04-30T06:43:16.351ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {} }
135082025-04-30T06:43:16.351ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
135092025-04-30T06:43:16.353ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "cf7c5c4d-74ac-4f4b-8fc0-e415e3db782e", "content-length": "723", "date": "Wed, 30 Apr 2025 06:43:16 GMT"} })
135102025-04-30T06:43:16.353ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/1/component/sp/caboose?firmware_slot=0
135112025-04-30T06:43:16.356ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
135122025-04-30T06:43:16.363ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
135132025-04-30T06:43:16.364ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
135142025-04-30T06:43:16.364ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {} }
135152025-04-30T06:43:16.364ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
135162025-04-30T06:43:16.369ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
135172025-04-30T06:43:16.369ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {} }
135182025-04-30T06:43:16.369ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
135192025-04-30T06:43:16.370ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "32f084f4-7688-4e93-bae8-95e44e845588", "content-length": "109", "date": "Wed, 30 Apr 2025 06:43:16 GMT"} })
135202025-04-30T06:43:16.370ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/1/component/sp/caboose?firmware_slot=1
135212025-04-30T06:43:16.375ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "bf3df512-20e6-4913-8078-19ed7a35eba4", "content-length": "109", "date": "Wed, 30 Apr 2025 06:43:16 GMT"} })
135222025-04-30T06:43:16.375ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/1/component/rot/caboose?firmware_slot=0
135232025-04-30T06:43:16.388ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "19495e87-dd99-492c-9a2d-696c0fec66b1", "content-length": "169", "date": "Wed, 30 Apr 2025 06:43:16 GMT"} })
135242025-04-30T06:43:16.389ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/1/component/rot/caboose?firmware_slot=1
135252025-04-30T06:43:16.393ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "89ed90b0-0fda-4758-a6b5-a0e3bfcfcd92", "content-length": "169", "date": "Wed, 30 Apr 2025 06:43:16 GMT"} })
135262025-04-30T06:43:16.393ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/1/component/stage0/caboose?firmware_slot=0
135272025-04-30T06:43:16.396ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "4b4af249-b4fb-4525-9b43-968c1af3ee9d", "content-length": "178", "date": "Wed, 30 Apr 2025 06:43:16 GMT"} })
135282025-04-30T06:43:16.396ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/1/component/stage0/caboose?firmware_slot=1
135292025-04-30T06:43:16.413ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "8073c874-22ec-4e20-9072-c7a9c4d8f57e", "content-length": "178", "date": "Wed, 30 Apr 2025 06:43:16 GMT"} })
135302025-04-30T06:43:16.413ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/1/component/rot/cmpa
135312025-04-30T06:43:16.414ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "902af99e-c51f-4aa0-9662-8a059fb44c89", "content-length": "702", "date": "Wed, 30 Apr 2025 06:43:16 GMT"} })
135322025-04-30T06:43:16.414ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/1/component/rot/cfpa
135332025-04-30T06:43:16.415ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "30c07929-1197-4df2-b9f1-849e36044ed8", "content-length": "727", "date": "Wed, 30 Apr 2025 06:43:16 GMT"} })
135342025-04-30T06:43:16.415ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/1/component/rot/cfpa
135352025-04-30T06:43:16.416ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "ed2b94bd-8119-4776-8845-fa6de9482d50", "content-length": "729", "date": "Wed, 30 Apr 2025 06:43:16 GMT"} })
135362025-04-30T06:43:16.416ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/1/component/rot/cfpa
135372025-04-30T06:43:16.417ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "8a309d4f-c966-4cc7-8339-d2af4cf9a459", "content-length": "728", "date": "Wed, 30 Apr 2025 06:43:16 GMT"} })
135382025-04-30T06:43:16.418ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
135392025-04-30T06:43:16.419ZTRACtest_snapshot_basic (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
135402025-04-30T06:43:16.419ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
135412025-04-30T06:43:16.419ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
135422025-04-30T06:43:16.420ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
135432025-04-30T06:43:16.426ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
135442025-04-30T06:43:16.438ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
135452025-04-30T06:43:16.438ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
135462025-04-30T06:43:16.438ZTRACtest_snapshot_basic (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
135472025-04-30T06:43:16.438ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
135482025-04-30T06:43:16.438ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
135492025-04-30T06:43:16.439ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
135502025-04-30T06:43:16.443ZDEBGtest_snapshot_basic: attempt to fetch latest inventory collection result = Ok("found")
135512025-04-30T06:43:16.443ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
135522025-04-30T06:43:16.449ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
135532025-04-30T06:43:16.449ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
135542025-04-30T06:43:16.449ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
135552025-04-30T06:43:16.450ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
135562025-04-30T06:43:16.509ZDEBGtest_snapshot_basic (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:41923
135572025-04-30T06:43:16.509ZDEBGtest_snapshot_basic (BackgroundTasks): client request SledAgent = http://[::1]:41923 background_task = inventory_collection body = None method = GET uri = http://[::1]:41923/inventory
135582025-04-30T06:43:16.509ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
135592025-04-30T06:43:16.509ZTRACtest_snapshot_basic (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
135602025-04-30T06:43:16.510ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
135612025-04-30T06:43:16.510ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {} }
135622025-04-30T06:43:16.510ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
135632025-04-30T06:43:16.511ZINFOtest_snapshot_basic (dropshot): accepted connection local_addr = [::1]:41923 remote_addr = [::1]:45960 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
135642025-04-30T06:43:16.512ZTRACtest_snapshot_basic (dropshot): incoming request local_addr = [::1]:41923 method = GET remote_addr = [::1]:45960 req_id = 85f572a0-9834-4181-9a28-33d12f246704 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
135652025-04-30T06:43:16.513ZINFOtest_snapshot_basic (dropshot): request completed latency_us = 781 local_addr = [::1]:41923 method = GET remote_addr = [::1]:45960 req_id = 85f572a0-9834-4181-9a28-33d12f246704 response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
135662025-04-30T06:43:16.514ZDEBGtest_snapshot_basic (BackgroundTasks): client response SledAgent = http://[::1]:41923 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:41923/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "85f572a0-9834-4181-9a28-33d12f246704", "content-length": "3141", "date": "Wed, 30 Apr 2025 06:43:16 GMT"} })
135672025-04-30T06:43:16.515ZDEBGtest_snapshot_basic (BackgroundTasks): begin collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
135682025-04-30T06:43:16.515ZDEBGtest_snapshot_basic (BackgroundTasks): end collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
135692025-04-30T06:43:16.515ZDEBGtest_snapshot_basic (BackgroundTasks): finished collection background_task = inventory_collection
135702025-04-30T06:43:16.517ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
135712025-04-30T06:43:16.517ZTRACtest_snapshot_basic (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
135722025-04-30T06:43:16.517ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
135732025-04-30T06:43:16.517ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
135742025-04-30T06:43:16.517ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
135752025-04-30T06:43:16.518ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
135762025-04-30T06:43:16.525ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
135772025-04-30T06:43:16.526ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
135782025-04-30T06:43:16.526ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {} }
135792025-04-30T06:43:16.526ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
135802025-04-30T06:43:16.534ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
135812025-04-30T06:43:16.542ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
135822025-04-30T06:43:16.542ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
135832025-04-30T06:43:16.542ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
135842025-04-30T06:43:16.543ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
135852025-04-30T06:43:16.545ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
135862025-04-30T06:43:16.545ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {} }
135872025-04-30T06:43:16.546ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
135882025-04-30T06:43:16.610ZWARNtest_snapshot_basic (datastore): Non-retryable transaction failure err = DatabaseError(UniqueViolation, "duplicate key value violates unique constraint \\"inv_caboose_pkey\\"") transaction = inventory_insert_collection
135892025-04-30T06:43:16.611ZWARNtest_snapshot_basic (BackgroundTasks): inventory collection failed background_task = inventory_collection error = failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint "inv_caboose_pkey"
135902025-04-30T06:43:16.611ZDEBGtest_snapshot_basic (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 476.995082ms iteration = 49 status = {"error":"failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint \\"inv_caboose_pkey\\""}
135912025-04-30T06:43:16.611ZDEBGtest_snapshot_basic (BackgroundTasks): activating background_task = inventory_collection iteration = 50 reason = Signaled
135922025-04-30T06:43:16.611ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
135932025-04-30T06:43:16.611ZTRACtest_snapshot_basic (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
135942025-04-30T06:43:16.611ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
135952025-04-30T06:43:16.611ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
135962025-04-30T06:43:16.612ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
135972025-04-30T06:43:16.619ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
135982025-04-30T06:43:16.626ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
135992025-04-30T06:43:16.626ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
136002025-04-30T06:43:16.627ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
136012025-04-30T06:43:16.627ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
136022025-04-30T06:43:16.628ZDEBGtest_snapshot_basic: attempt to fetch latest inventory collection result = Ok("found")
136032025-04-30T06:43:16.643ZDEBGtest_snapshot_basic (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = [(d1ad44e5-18dd-4193-a70e-bf9042f0e0e1, 0)]
136042025-04-30T06:43:16.643ZTRACtest_snapshot_basic (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
136052025-04-30T06:43:16.645ZTRACtest_snapshot_basic (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: 5d 20 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 ] ..........._mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:49476 req_id = 541d1cb4-d98b-43c3-b377-0a98947d938d
136062025-04-30T06:43:16.645ZDEBGtest_snapshot_basic (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 23840,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:49476 req_id = 541d1cb4-d98b-43c3-b377-0a98947d938d
136072025-04-30T06:43:16.645ZDEBGtest_snapshot_basic (store): query key key = _mgs._tcp
136082025-04-30T06:43:16.645ZDEBGtest_snapshot_basic (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
136092025-04-30T06:43:16.645ZDEBGtest_snapshot_basic (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:49476 query = LowerQuery { name: LowerName(Name("_mgs._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 41502, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = 541d1cb4-d98b-43c3-b377-0a98947d938d
136102025-04-30T06:43:16.646ZDEBGtest_snapshot_basic (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 41502, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 4261, tv_nsec: 445207048 } })
136112025-04-30T06:43:16.646ZTRACtest_snapshot_basic (DnsResolver): lookup_service_targets: looking up SRV target
136122025-04-30T06:43:16.647ZTRACtest_snapshot_basic (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: fa 43 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e .C..........-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:62286 req_id = 49a138ed-b923-4fc1-a7ca-008772d1e356
136132025-04-30T06:43:16.647ZDEBGtest_snapshot_basic (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 64067,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:62286 req_id = 49a138ed-b923-4fc1-a7ca-008772d1e356
136142025-04-30T06:43:16.647ZDEBGtest_snapshot_basic (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
136152025-04-30T06:43:16.647ZDEBGtest_snapshot_basic (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:62286 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = 49a138ed-b923-4fc1-a7ca-008772d1e356
136162025-04-30T06:43:16.709ZTRACtest_snapshot_basic (DnsResolver): lookup_all_socket_v6 srv dns_name = _clickhouse-admin-keeper._tcp.control-plane.oxide.internal
136172025-04-30T06:43:16.709ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
136182025-04-30T06:43:16.709ZTRACtest_snapshot_basic (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
136192025-04-30T06:43:16.709ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
136202025-04-30T06:43:16.709ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {} }
136212025-04-30T06:43:16.711ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
136222025-04-30T06:43:16.713ZTRACtest_snapshot_basic (dns): buffer buffer = Length: 87 (0x57) bytes\n0000: 73 05 01 00 00 01 00 00 00 00 00 01 18 5f 63 6c s............_cl\n0010: 69 63 6b 68 6f 75 73 65 2d 61 64 6d 69 6e 2d 6b ickhouse-admin-k\n0020: 65 65 70 65 72 04 5f 74 63 70 0d 63 6f 6e 74 72 eeper._tcp.contr\n0030: 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 69 ol-plane.oxide.i\n0040: 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 04 nternal..!....).\n0050: d0 00 00 00 00 00 00 ....... kind = dns peer_addr = [::1]:55999 req_id = 37c36aaa-219a-45f2-b2a0-6bfff99574de
136232025-04-30T06:43:16.713ZDEBGtest_snapshot_basic (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 29445,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 24,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 97,\n 100,\n 109,\n 105,\n 110,\n 45,\n 107,\n 101,\n 101,\n 112,\n 101,\n 114,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:55999 req_id = 37c36aaa-219a-45f2-b2a0-6bfff99574de
136242025-04-30T06:43:16.713ZDEBGtest_snapshot_basic (store): query key key = _clickhouse-admin-keeper._tcp
136252025-04-30T06:43:16.714ZERROtest_snapshot_basic (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 29445, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } NXDOMAIN: no records found for name: "_clickhouse-admin-keeper._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:55999 req_id = 37c36aaa-219a-45f2-b2a0-6bfff99574de
136262025-04-30T06:43:16.714ZDEBGtest_snapshot_basic (BackgroundTasks): begin collection background_task = inventory_collection
136272025-04-30T06:43:16.714ZDEBGtest_snapshot_basic (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:41502
136282025-04-30T06:43:16.714ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/ignition
136292025-04-30T06:43:16.716ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
136302025-04-30T06:43:16.731ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
136312025-04-30T06:43:16.731ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
136322025-04-30T06:43:16.732ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {} }
136332025-04-30T06:43:16.732ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
136342025-04-30T06:43:16.732ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "4722d9a5-91bb-404e-a06e-cb575bb99bbb", "content-length": "835", "date": "Wed, 30 Apr 2025 06:43:16 GMT"} })
136352025-04-30T06:43:16.733ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/0
136362025-04-30T06:43:16.738ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
136372025-04-30T06:43:16.738ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {} }
136382025-04-30T06:43:16.739ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
136392025-04-30T06:43:16.741ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "763cf7d2-34ef-45d2-9005-ce4ad1cb4b6e", "content-length": "734", "date": "Wed, 30 Apr 2025 06:43:16 GMT"} })
136402025-04-30T06:43:16.741ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/0/component/sp/caboose?firmware_slot=0
136412025-04-30T06:43:16.744ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "f89037fa-3e5d-4c2c-8c74-e027ed73e479", "content-length": "111", "date": "Wed, 30 Apr 2025 06:43:16 GMT"} })
136422025-04-30T06:43:16.744ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/0/component/sp/caboose?firmware_slot=1
136432025-04-30T06:43:16.748ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "691252dc-f135-4ace-9f31-a28e7b3b154c", "content-length": "111", "date": "Wed, 30 Apr 2025 06:43:16 GMT"} })
136442025-04-30T06:43:16.748ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/0/component/rot/caboose?firmware_slot=0
136452025-04-30T06:43:16.751ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "dcfb01a2-521f-4b1b-b515-a6385d98c5f8", "content-length": "170", "date": "Wed, 30 Apr 2025 06:43:16 GMT"} })
136462025-04-30T06:43:16.751ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/0/component/rot/caboose?firmware_slot=1
136472025-04-30T06:43:16.755ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "415cf896-cf67-4ef8-9c44-cbaee13d6288", "content-length": "170", "date": "Wed, 30 Apr 2025 06:43:16 GMT"} })
136482025-04-30T06:43:16.755ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/0/component/stage0/caboose?firmware_slot=0
136492025-04-30T06:43:16.757ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "7efb3a1b-7da7-4de4-9c23-70a726f42fbc", "content-length": "179", "date": "Wed, 30 Apr 2025 06:43:16 GMT"} })
136502025-04-30T06:43:16.758ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/0/component/stage0/caboose?firmware_slot=1
136512025-04-30T06:43:16.760ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "b29c73b5-3214-4f13-933b-42a4f795070d", "content-length": "179", "date": "Wed, 30 Apr 2025 06:43:16 GMT"} })
136522025-04-30T06:43:16.761ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/0/component/rot/cmpa
136532025-04-30T06:43:16.762ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "f7e38a1d-be9b-42a0-9ea8-dbb077422114", "content-length": "702", "date": "Wed, 30 Apr 2025 06:43:16 GMT"} })
136542025-04-30T06:43:16.762ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/0/component/rot/cfpa
136552025-04-30T06:43:16.763ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "5e5f6cc1-f3b0-4905-876a-21058a735ba3", "content-length": "727", "date": "Wed, 30 Apr 2025 06:43:16 GMT"} })
136562025-04-30T06:43:16.764ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/0/component/rot/cfpa
136572025-04-30T06:43:16.765ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "9b66097e-e865-4ec2-85b9-424aaa2a0cbd", "content-length": "729", "date": "Wed, 30 Apr 2025 06:43:16 GMT"} })
136582025-04-30T06:43:16.765ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/0/component/rot/cfpa
136592025-04-30T06:43:16.770ZDEBGtest_snapshot_basic: attempt to fetch latest inventory collection result = Ok("found")
136602025-04-30T06:43:16.772ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "cce8df4a-6e3e-44dd-9bde-40f2a561eb3c", "content-length": "728", "date": "Wed, 30 Apr 2025 06:43:16 GMT"} })
136612025-04-30T06:43:16.772ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/1
136622025-04-30T06:43:16.775ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "90b2a8da-27be-4b2f-8e8d-3f7dc1e8b216", "content-length": "734", "date": "Wed, 30 Apr 2025 06:43:16 GMT"} })
136632025-04-30T06:43:16.775ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/1/component/sp/caboose?firmware_slot=0
136642025-04-30T06:43:16.778ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "c0ea6917-aa93-47f1-8162-3707c852514d", "content-length": "111", "date": "Wed, 30 Apr 2025 06:43:16 GMT"} })
136652025-04-30T06:43:16.779ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/1/component/sp/caboose?firmware_slot=1
136662025-04-30T06:43:16.781ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "4faf3be2-c941-4afb-a9d6-0648d5924808", "content-length": "111", "date": "Wed, 30 Apr 2025 06:43:16 GMT"} })
136672025-04-30T06:43:16.781ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/1/component/rot/caboose?firmware_slot=0
136682025-04-30T06:43:16.784ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "3d3d50f3-bde2-43de-8b8f-0cb51eaaf8e6", "content-length": "170", "date": "Wed, 30 Apr 2025 06:43:16 GMT"} })
136692025-04-30T06:43:16.785ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/1/component/rot/caboose?firmware_slot=1
136702025-04-30T06:43:16.787ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "c911b80a-3118-49a0-9681-8791f0dfaf71", "content-length": "170", "date": "Wed, 30 Apr 2025 06:43:16 GMT"} })
136712025-04-30T06:43:16.787ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/1/component/stage0/caboose?firmware_slot=0
136722025-04-30T06:43:16.789ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "de7d3f78-c047-4c19-8b1a-e79ecb51fdf8", "content-length": "179", "date": "Wed, 30 Apr 2025 06:43:16 GMT"} })
136732025-04-30T06:43:16.790ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/1/component/stage0/caboose?firmware_slot=1
136742025-04-30T06:43:16.794ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "802920c8-d9c4-43d4-a36a-4102e9dc687f", "content-length": "179", "date": "Wed, 30 Apr 2025 06:43:16 GMT"} })
136752025-04-30T06:43:16.794ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/1/component/rot/cmpa
136762025-04-30T06:43:16.795ZDEBGtest_snapshot_basic (collection-task): sent timer-based collection request to the collection task collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
136772025-04-30T06:43:16.795ZDEBGtest_snapshot_basic (collection-task): collecting from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
136782025-04-30T06:43:16.800ZTRACtest_snapshot_basic (collection-task): sent collection request to producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
136792025-04-30T06:43:16.801ZDEBGtest_snapshot_basic (collection-task): collected results from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 n_results = 1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
136802025-04-30T06:43:16.801ZTRACtest_snapshot_basic (collection-task): forwarded results to main collection loop collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
136812025-04-30T06:43:16.801ZTRACtest_snapshot_basic (collection-task): top of inner collection loop, waiting for next request collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
136822025-04-30T06:43:16.802ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "617c0bcf-6ee5-49ac-8bef-850f4f16afcb", "content-length": "702", "date": "Wed, 30 Apr 2025 06:43:16 GMT"} })
136832025-04-30T06:43:16.802ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/1/component/rot/cfpa
136842025-04-30T06:43:16.804ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "40003a8d-557c-4aef-aded-19862ce46d03", "content-length": "727", "date": "Wed, 30 Apr 2025 06:43:16 GMT"} })
136852025-04-30T06:43:16.804ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/1/component/rot/cfpa
136862025-04-30T06:43:16.805ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "314dceea-5582-4c4d-bb00-8f1140325b7a", "content-length": "729", "date": "Wed, 30 Apr 2025 06:43:16 GMT"} })
136872025-04-30T06:43:16.806ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/1/component/rot/cfpa
136882025-04-30T06:43:16.807ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "18ce99b7-2ef3-4354-8731-159c4392f1db", "content-length": "728", "date": "Wed, 30 Apr 2025 06:43:16 GMT"} })
136892025-04-30T06:43:16.807ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/0
136902025-04-30T06:43:16.808ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "5588115c-7ea5-471c-ad4c-782fd5264bdd", "content-length": "723", "date": "Wed, 30 Apr 2025 06:43:16 GMT"} })
136912025-04-30T06:43:16.808ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/0/component/sp/caboose?firmware_slot=0
136922025-04-30T06:43:16.810ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "dc4920a2-542d-4adb-ba75-5eb2c36a32a3", "content-length": "109", "date": "Wed, 30 Apr 2025 06:43:16 GMT"} })
136932025-04-30T06:43:16.810ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/0/component/sp/caboose?firmware_slot=1
136942025-04-30T06:43:16.812ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "dfef2713-7639-4091-990d-058a81429d25", "content-length": "109", "date": "Wed, 30 Apr 2025 06:43:16 GMT"} })
136952025-04-30T06:43:16.813ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/0/component/rot/caboose?firmware_slot=0
136962025-04-30T06:43:16.814ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "bc1690d1-d04d-4d2d-823f-392bd559bba8", "content-length": "169", "date": "Wed, 30 Apr 2025 06:43:16 GMT"} })
136972025-04-30T06:43:16.814ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/0/component/rot/caboose?firmware_slot=1
136982025-04-30T06:43:16.817ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "3f3dff52-238b-4999-8ec0-b98510c34430", "content-length": "169", "date": "Wed, 30 Apr 2025 06:43:16 GMT"} })
136992025-04-30T06:43:16.818ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/0/component/stage0/caboose?firmware_slot=0
137002025-04-30T06:43:16.820ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "3466c670-0614-4da4-bb78-1ae3db20a22b", "content-length": "178", "date": "Wed, 30 Apr 2025 06:43:16 GMT"} })
137012025-04-30T06:43:16.820ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/0/component/stage0/caboose?firmware_slot=1
137022025-04-30T06:43:16.820ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
137032025-04-30T06:43:16.820ZTRACtest_snapshot_basic (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
137042025-04-30T06:43:16.821ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
137052025-04-30T06:43:16.821ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {} }
137062025-04-30T06:43:16.821ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
137072025-04-30T06:43:16.824ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
137082025-04-30T06:43:16.832ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
137092025-04-30T06:43:16.832ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
137102025-04-30T06:43:16.832ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {} }
137112025-04-30T06:43:16.833ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
137122025-04-30T06:43:16.834ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "5b063c1b-97cb-41e7-9b60-7356269cbdd7", "content-length": "178", "date": "Wed, 30 Apr 2025 06:43:16 GMT"} })
137132025-04-30T06:43:16.835ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/0/component/rot/cmpa
137142025-04-30T06:43:16.838ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
137152025-04-30T06:43:16.838ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {} }
137162025-04-30T06:43:16.838ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
137172025-04-30T06:43:16.839ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "dedca48e-5daa-47a3-b05c-5bd3b9a67bae", "content-length": "702", "date": "Wed, 30 Apr 2025 06:43:16 GMT"} })
137182025-04-30T06:43:16.839ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/0/component/rot/cfpa
137192025-04-30T06:43:16.840ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "009a1c47-184e-464d-aab5-098760a5c74c", "content-length": "727", "date": "Wed, 30 Apr 2025 06:43:16 GMT"} })
137202025-04-30T06:43:16.840ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/0/component/rot/cfpa
137212025-04-30T06:43:16.842ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "460c7528-6cb1-4ae3-a89d-82a043963400", "content-length": "729", "date": "Wed, 30 Apr 2025 06:43:16 GMT"} })
137222025-04-30T06:43:16.842ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/0/component/rot/cfpa
137232025-04-30T06:43:16.843ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "0c5f8970-442b-4fc3-bdcf-fb082331a5fd", "content-length": "728", "date": "Wed, 30 Apr 2025 06:43:16 GMT"} })
137242025-04-30T06:43:16.843ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/1
137252025-04-30T06:43:16.846ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "bebbbb13-caa7-4902-9dd9-ebae9b676b61", "content-length": "723", "date": "Wed, 30 Apr 2025 06:43:16 GMT"} })
137262025-04-30T06:43:16.846ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/1/component/sp/caboose?firmware_slot=0
137272025-04-30T06:43:16.849ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "8ff898df-f1f3-43ea-8168-4ad64ed5926a", "content-length": "109", "date": "Wed, 30 Apr 2025 06:43:16 GMT"} })
137282025-04-30T06:43:16.849ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/1/component/sp/caboose?firmware_slot=1
137292025-04-30T06:43:16.854ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "a26a4365-6e0d-42c9-9caf-384dd8e04a3d", "content-length": "109", "date": "Wed, 30 Apr 2025 06:43:16 GMT"} })
137302025-04-30T06:43:16.855ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/1/component/rot/caboose?firmware_slot=0
137312025-04-30T06:43:16.860ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "b9724357-c467-4e43-a3c0-eabf1a8e1675", "content-length": "169", "date": "Wed, 30 Apr 2025 06:43:16 GMT"} })
137322025-04-30T06:43:16.860ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/1/component/rot/caboose?firmware_slot=1
137332025-04-30T06:43:16.862ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "69e10969-e8ca-498a-8821-a1c2b8ccbd0c", "content-length": "169", "date": "Wed, 30 Apr 2025 06:43:16 GMT"} })
137342025-04-30T06:43:16.862ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/1/component/stage0/caboose?firmware_slot=0
137352025-04-30T06:43:16.867ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "dcb14265-9ddd-4776-884a-8265aeb76c3b", "content-length": "178", "date": "Wed, 30 Apr 2025 06:43:16 GMT"} })
137362025-04-30T06:43:16.867ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/1/component/stage0/caboose?firmware_slot=1
137372025-04-30T06:43:16.874ZDEBGtest_snapshot_basic: attempt to fetch latest inventory collection result = Ok("found")
137382025-04-30T06:43:16.876ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "27dafc01-0049-4bf6-9c0c-75b67a68d6c8", "content-length": "178", "date": "Wed, 30 Apr 2025 06:43:16 GMT"} })
137392025-04-30T06:43:16.876ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/1/component/rot/cmpa
137402025-04-30T06:43:16.878ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "671059c2-efff-4570-87c5-8114ed8c8e73", "content-length": "702", "date": "Wed, 30 Apr 2025 06:43:16 GMT"} })
137412025-04-30T06:43:16.878ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/1/component/rot/cfpa
137422025-04-30T06:43:16.880ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "e1492a01-6a01-4952-956e-4568ca31341b", "content-length": "727", "date": "Wed, 30 Apr 2025 06:43:16 GMT"} })
137432025-04-30T06:43:16.880ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/1/component/rot/cfpa
137442025-04-30T06:43:16.881ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "be76c135-6723-4fb4-8baa-27f2f5755d9c", "content-length": "729", "date": "Wed, 30 Apr 2025 06:43:16 GMT"} })
137452025-04-30T06:43:16.881ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/1/component/rot/cfpa
137462025-04-30T06:43:16.883ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "a9867607-e646-45f1-8ab0-e04859dc8aae", "content-length": "728", "date": "Wed, 30 Apr 2025 06:43:16 GMT"} })
137472025-04-30T06:43:16.883ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
137482025-04-30T06:43:16.883ZTRACtest_snapshot_basic (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
137492025-04-30T06:43:16.883ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
137502025-04-30T06:43:16.883ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
137512025-04-30T06:43:16.884ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
137522025-04-30T06:43:16.889ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
137532025-04-30T06:43:16.895ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
137542025-04-30T06:43:16.895ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
137552025-04-30T06:43:16.895ZTRACtest_snapshot_basic (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
137562025-04-30T06:43:16.895ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
137572025-04-30T06:43:16.895ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
137582025-04-30T06:43:16.896ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
137592025-04-30T06:43:16.899ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
137602025-04-30T06:43:16.909ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
137612025-04-30T06:43:16.909ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
137622025-04-30T06:43:16.909ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
137632025-04-30T06:43:16.910ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
137642025-04-30T06:43:17.009ZDEBGtest_snapshot_basic (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:41923
137652025-04-30T06:43:17.009ZDEBGtest_snapshot_basic (BackgroundTasks): client request SledAgent = http://[::1]:41923 background_task = inventory_collection body = None method = GET uri = http://[::1]:41923/inventory
137662025-04-30T06:43:17.010ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
137672025-04-30T06:43:17.010ZTRACtest_snapshot_basic (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
137682025-04-30T06:43:17.010ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
137692025-04-30T06:43:17.010ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {} }
137702025-04-30T06:43:17.012ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
137712025-04-30T06:43:17.012ZINFOtest_snapshot_basic (dropshot): accepted connection local_addr = [::1]:41923 remote_addr = [::1]:64099 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
137722025-04-30T06:43:17.015ZTRACtest_snapshot_basic (dropshot): incoming request local_addr = [::1]:41923 method = GET remote_addr = [::1]:64099 req_id = 3d0370a9-dbd3-42b8-afcc-432d0feb044b sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
137732025-04-30T06:43:17.529ZINFOtest_snapshot_basic (dropshot): request completed latency_us = 513528 local_addr = [::1]:41923 method = GET remote_addr = [::1]:64099 req_id = 3d0370a9-dbd3-42b8-afcc-432d0feb044b response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
137742025-04-30T06:43:17.529ZINFOtest_snapshot_basic (dropshot): accepted connection kind = pantry local_addr = [::1]:62733 remote_addr = [::1]:36605 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
137752025-04-30T06:43:17.529ZINFOtest_snapshot_basic (dropshot): accepted connection kind = pantry local_addr = [::1]:62733 remote_addr = [::1]:40226 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
137762025-04-30T06:43:17.529ZINFOtest_snapshot_basic (dropshot): accepted connection kind = pantry local_addr = [::1]:62733 remote_addr = [::1]:44049 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
137772025-04-30T06:43:17.529ZINFOtest_snapshot_basic (dropshot): accepted connection kind = pantry local_addr = [::1]:62733 remote_addr = [::1]:52030 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
137782025-04-30T06:43:17.529ZINFOtest_snapshot_basic (dropshot): accepted connection kind = pantry local_addr = [::1]:62733 remote_addr = [::1]:45238 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
137792025-04-30T06:43:17.529ZINFOtest_snapshot_basic (dropshot): accepted connection kind = pantry local_addr = [::1]:62733 remote_addr = [::1]:38900 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
137802025-04-30T06:43:17.529ZINFOtest_snapshot_basic (dropshot): accepted connection kind = pantry local_addr = [::1]:62733 remote_addr = [::1]:53195 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
137812025-04-30T06:43:17.530ZDEBGtest_snapshot_basic (results-sink): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
137822025-04-30T06:43:17.530ZDEBGtest_snapshot_basic (results-sink-cluster): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
137832025-04-30T06:43:17.530ZTRACtest_snapshot_basic (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal
137842025-04-30T06:43:17.530ZTRACtest_snapshot_basic (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal
137852025-04-30T06:43:17.530ZTRACtest_snapshot_basic (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal
137862025-04-30T06:43:17.531ZDEBGtest_snapshot_basic (BackgroundTasks): client response SledAgent = http://[::1]:41923 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:41923/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "3d0370a9-dbd3-42b8-afcc-432d0feb044b", "content-length": "3141", "date": "Wed, 30 Apr 2025 06:43:17 GMT"} })
137872025-04-30T06:43:17.533ZDEBGtest_snapshot_basic (BackgroundTasks): begin collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
137882025-04-30T06:43:17.533ZDEBGtest_snapshot_basic (BackgroundTasks): end collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
137892025-04-30T06:43:17.533ZDEBGtest_snapshot_basic (BackgroundTasks): finished collection background_task = inventory_collection
137902025-04-30T06:43:17.536ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
137912025-04-30T06:43:17.536ZTRACtest_snapshot_basic (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
137922025-04-30T06:43:17.536ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
137932025-04-30T06:43:17.536ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
137942025-04-30T06:43:17.536ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
137952025-04-30T06:43:17.537ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
137962025-04-30T06:43:17.544ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
137972025-04-30T06:43:17.544ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
137982025-04-30T06:43:17.545ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {} }
137992025-04-30T06:43:17.545ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
138002025-04-30T06:43:17.547ZDEBGtest_snapshot_basic (clickhouse-client): unrolling 1 total samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 6ad06c6b-9a6e-4ed1-b3f4-5d2e9da3f8a7
138012025-04-30T06:43:17.549ZTRACtest_snapshot_basic (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 6ad06c6b-9a6e-4ed1-b3f4-5d2e9da3f8a7 n_columns = 4 n_rows = 2 sql = INSERT INTO oximeter.fields_string FORMAT Native
138022025-04-30T06:43:17.551ZTRACtest_snapshot_basic (dropshot): incoming request kind = pantry local_addr = [::1]:62733 method = GET remote_addr = [::1]:36605 req_id = fb9cb123-1a6e-4d31-857c-761b415b7430 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /crucible/pantry/0
138032025-04-30T06:43:17.551ZTRACtest_snapshot_basic (dropshot): incoming request kind = pantry local_addr = [::1]:62733 method = GET remote_addr = [::1]:40226 req_id = b1d3ba9c-c52c-4d72-afaf-e1e4b8108f37 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /crucible/pantry/0
138042025-04-30T06:43:17.552ZTRACtest_snapshot_basic (dropshot): incoming request kind = pantry local_addr = [::1]:62733 method = GET remote_addr = [::1]:44049 req_id = 76d034a1-b765-40fe-a14a-b5e55ac9d4e5 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /crucible/pantry/0
138052025-04-30T06:43:17.552ZTRACtest_snapshot_basic (dropshot): incoming request kind = pantry local_addr = [::1]:62733 method = GET remote_addr = [::1]:52030 req_id = 9580e72c-7c09-4f0f-9a68-66f16e7d5282 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /crucible/pantry/0
138062025-04-30T06:43:17.552ZTRACtest_snapshot_basic (dropshot): incoming request kind = pantry local_addr = [::1]:62733 method = GET remote_addr = [::1]:45238 req_id = 5aed57eb-273a-4add-813e-5d79d658214e sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /crucible/pantry/0
138072025-04-30T06:43:17.553ZTRACtest_snapshot_basic (dropshot): incoming request kind = pantry local_addr = [::1]:62733 method = GET remote_addr = [::1]:38900 req_id = 71a4df0d-b9de-421f-be4b-d72cce85acf0 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /crucible/pantry/0
138082025-04-30T06:43:17.553ZTRACtest_snapshot_basic (dropshot): incoming request kind = pantry local_addr = [::1]:62733 method = GET remote_addr = [::1]:53195 req_id = dc4f847f-a2e8-461c-81de-c61aa888097c sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /crucible/pantry/0
138092025-04-30T06:43:17.553ZTRACtest_snapshot_basic (dns): buffer buffer = Length: 72 (0x48) bytes\n0000: 6c 23 01 00 00 01 00 00 00 00 00 01 09 5f 64 65 l#..........._de\n0010: 6e 64 72 69 74 65 04 5f 74 63 70 0d 63 6f 6e 74 ndrite._tcp.cont\n0020: 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 rol-plane.oxide.\n0030: 69 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 internal..!....)\n0040: 04 d0 00 00 00 00 00 00 ........ kind = dns peer_addr = [::1]:52568 req_id = b8e29e23-ac81-467e-87db-854d61ae3345
138102025-04-30T06:43:17.554ZDEBGtest_snapshot_basic (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 27683,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_dendrite._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_dendrite._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 9,\n 95,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:52568 req_id = b8e29e23-ac81-467e-87db-854d61ae3345
138112025-04-30T06:43:17.554ZDEBGtest_snapshot_basic (store): query key key = _dendrite._tcp
138122025-04-30T06:43:17.555ZDEBGtest_snapshot_basic (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
138132025-04-30T06:43:17.555ZDEBGtest_snapshot_basic (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:52568 query = LowerQuery { name: LowerName(Name("_dendrite._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 32976, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = b8e29e23-ac81-467e-87db-854d61ae3345
138142025-04-30T06:43:17.555ZTRACtest_snapshot_basic (dns): buffer buffer = Length: 72 (0x48) bytes\n0000: f4 ce 01 00 00 01 00 00 00 00 00 01 09 5f 64 65 ............._de\n0010: 6e 64 72 69 74 65 04 5f 74 63 70 0d 63 6f 6e 74 ndrite._tcp.cont\n0020: 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 rol-plane.oxide.\n0030: 69 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 internal..!....)\n0040: 04 d0 00 00 00 00 00 00 ........ kind = dns peer_addr = [::1]:57793 req_id = f84ee010-663d-45d5-b8e7-d2adead9c2e0
138152025-04-30T06:43:17.556ZDEBGtest_snapshot_basic (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 62670,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_dendrite._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_dendrite._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 9,\n 95,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:57793 req_id = f84ee010-663d-45d5-b8e7-d2adead9c2e0
138162025-04-30T06:43:17.556ZDEBGtest_snapshot_basic (store): query key key = _dendrite._tcp
138172025-04-30T06:43:17.556ZDEBGtest_snapshot_basic (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
138182025-04-30T06:43:17.557ZDEBGtest_snapshot_basic (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:57793 query = LowerQuery { name: LowerName(Name("_dendrite._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 32976, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = f84ee010-663d-45d5-b8e7-d2adead9c2e0
138192025-04-30T06:43:17.557ZTRACtest_snapshot_basic (dns): buffer buffer = Length: 72 (0x48) bytes\n0000: 67 a9 01 00 00 01 00 00 00 00 00 01 09 5f 64 65 g............_de\n0010: 6e 64 72 69 74 65 04 5f 74 63 70 0d 63 6f 6e 74 ndrite._tcp.cont\n0020: 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 rol-plane.oxide.\n0030: 69 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 internal..!....)\n0040: 04 d0 00 00 00 00 00 00 ........ kind = dns peer_addr = [::1]:51170 req_id = aa1ba8d7-dd5f-4306-92f2-f034bcb7bb19
138202025-04-30T06:43:17.557ZDEBGtest_snapshot_basic (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 26537,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_dendrite._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_dendrite._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 9,\n 95,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:51170 req_id = aa1ba8d7-dd5f-4306-92f2-f034bcb7bb19
138212025-04-30T06:43:17.557ZDEBGtest_snapshot_basic (store): query key key = _dendrite._tcp
138222025-04-30T06:43:17.557ZDEBGtest_snapshot_basic (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
138232025-04-30T06:43:17.561ZDEBGtest_snapshot_basic (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:51170 query = LowerQuery { name: LowerName(Name("_dendrite._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 32976, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = aa1ba8d7-dd5f-4306-92f2-f034bcb7bb19
138242025-04-30T06:43:17.563ZINFOtest_snapshot_basic (dropshot): request completed kind = pantry latency_us = 11584 local_addr = [::1]:62733 method = GET remote_addr = [::1]:36605 req_id = fb9cb123-1a6e-4d31-857c-761b415b7430 response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /crucible/pantry/0
138252025-04-30T06:43:17.563ZINFOtest_snapshot_basic (dropshot): request completed kind = pantry latency_us = 11420 local_addr = [::1]:62733 method = GET remote_addr = [::1]:40226 req_id = b1d3ba9c-c52c-4d72-afaf-e1e4b8108f37 response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /crucible/pantry/0
138262025-04-30T06:43:17.563ZINFOtest_snapshot_basic (dropshot): request completed kind = pantry latency_us = 10953 local_addr = [::1]:62733 method = GET remote_addr = [::1]:44049 req_id = 76d034a1-b765-40fe-a14a-b5e55ac9d4e5 response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /crucible/pantry/0
138272025-04-30T06:43:17.563ZINFOtest_snapshot_basic (dropshot): request completed kind = pantry latency_us = 10784 local_addr = [::1]:62733 method = GET remote_addr = [::1]:52030 req_id = 9580e72c-7c09-4f0f-9a68-66f16e7d5282 response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /crucible/pantry/0
138282025-04-30T06:43:17.565ZINFOtest_snapshot_basic (dropshot): request completed kind = pantry latency_us = 12619 local_addr = [::1]:62733 method = GET remote_addr = [::1]:45238 req_id = 5aed57eb-273a-4add-813e-5d79d658214e response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /crucible/pantry/0
138292025-04-30T06:43:17.565ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
138302025-04-30T06:43:17.575ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
138312025-04-30T06:43:17.575ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
138322025-04-30T06:43:17.575ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
138332025-04-30T06:43:17.578ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
138342025-04-30T06:43:17.578ZINFOtest_snapshot_basic (dropshot): request completed kind = pantry latency_us = 25250 local_addr = [::1]:62733 method = GET remote_addr = [::1]:38900 req_id = 71a4df0d-b9de-421f-be4b-d72cce85acf0 response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /crucible/pantry/0
138352025-04-30T06:43:17.578ZINFOtest_snapshot_basic (dropshot): request completed kind = pantry latency_us = 25150 local_addr = [::1]:62733 method = GET remote_addr = [::1]:53195 req_id = dc4f847f-a2e8-461c-81de-c61aa888097c response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /crucible/pantry/0
138362025-04-30T06:43:17.589ZDEBGtest_snapshot_basic (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 32976, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 4262, tv_nsec: 388706659 } })
138372025-04-30T06:43:17.589ZTRACtest_snapshot_basic (DnsResolver): lookup_service_targets: looking up SRV target
138382025-04-30T06:43:17.590ZDEBGtest_snapshot_basic (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 32976, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 4262, tv_nsec: 389265632 } })
138392025-04-30T06:43:17.590ZTRACtest_snapshot_basic (DnsResolver): lookup_service_targets: looking up SRV target
138402025-04-30T06:43:17.590ZDEBGtest_snapshot_basic (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 32976, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 4262, tv_nsec: 389580259 } })
138412025-04-30T06:43:17.590ZTRACtest_snapshot_basic (DnsResolver): lookup_service_targets: looking up SRV target
138422025-04-30T06:43:17.592ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
138432025-04-30T06:43:17.592ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {} }
138442025-04-30T06:43:17.592ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
138452025-04-30T06:43:17.594ZDEBGtest_snapshot_basic (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 6ad06c6b-9a6e-4ed1-b3f4-5d2e9da3f8a7 n_rows = 2 table_name = fields_string
138462025-04-30T06:43:17.594ZTRACtest_snapshot_basic (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 6ad06c6b-9a6e-4ed1-b3f4-5d2e9da3f8a7 n_columns = 4 n_rows = 1 sql = INSERT INTO oximeter.measurements_i64 FORMAT Native
138472025-04-30T06:43:17.605ZTRACtest_snapshot_basic (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: 15 15 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e ............-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:55255 req_id = 2e153740-1c70-434b-9a41-3f06b391c35c
138482025-04-30T06:43:17.605ZDEBGtest_snapshot_basic (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 5397,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:55255 req_id = 2e153740-1c70-434b-9a41-3f06b391c35c
138492025-04-30T06:43:17.605ZDEBGtest_snapshot_basic (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
138502025-04-30T06:43:17.606ZDEBGtest_snapshot_basic (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:55255 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = 2e153740-1c70-434b-9a41-3f06b391c35c
138512025-04-30T06:43:17.606ZTRACtest_snapshot_basic (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: da 60 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e .`..........-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:63707 req_id = e2d08587-a079-4d8c-aa26-6772db6adc51
138522025-04-30T06:43:17.606ZDEBGtest_snapshot_basic (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 55904,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:63707 req_id = e2d08587-a079-4d8c-aa26-6772db6adc51
138532025-04-30T06:43:17.607ZDEBGtest_snapshot_basic (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
138542025-04-30T06:43:17.608ZDEBGtest_snapshot_basic (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:63707 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = e2d08587-a079-4d8c-aa26-6772db6adc51
138552025-04-30T06:43:17.608ZTRACtest_snapshot_basic (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: bd 34 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e .4..........-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:50706 req_id = 6c68c1b2-fc47-4218-9df1-e62e78ecdc0d
138562025-04-30T06:43:17.609ZDEBGtest_snapshot_basic (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 48436,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:50706 req_id = 6c68c1b2-fc47-4218-9df1-e62e78ecdc0d
138572025-04-30T06:43:17.609ZDEBGtest_snapshot_basic (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
138582025-04-30T06:43:17.609ZDEBGtest_snapshot_basic (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:50706 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = 6c68c1b2-fc47-4218-9df1-e62e78ecdc0d
138592025-04-30T06:43:17.613ZINFOtest_snapshot_basic (BackgroundTasks): Determining switch slots managed by switch zones background_task = nat_v4_garbage_collector
138602025-04-30T06:43:17.694ZINFOtest_snapshot_basic (BackgroundTasks): determining switch slot managed by dendrite zone background_task = nat_v4_garbage_collector zone_address = ::1
138612025-04-30T06:43:17.694ZDEBGtest_snapshot_basic (MgsClient): client request background_task = nat_v4_garbage_collector body = None method = GET uri = http://[::1]:12225/local/switch-id
138622025-04-30T06:43:17.694ZDEBGtest_snapshot_basic (MgsClient): client response background_task = nat_v4_garbage_collector result = Err(reqwest::Error { kind: Request, url: "http://[::1]:12225/local/switch-id", source: hyper_util::client::legacy::Error(Connect, ConnectError("tcp connect error", Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) })
138632025-04-30T06:43:17.694ZWARNtest_snapshot_basic (BackgroundTasks): failed to identify switch slot for dendrite background_task = nat_v4_garbage_collector reason = Communication Error: error sending request for url (http://[::1]:12225/local/switch-id): client error (Connect): tcp connect error: Connection refused (os error 146): Connection refused (os error 146) zone_address = ::1
138642025-04-30T06:43:17.695ZWARNtest_snapshot_basic (BackgroundTasks): Failed to map switch zone addr: Communication Error: error sending request for url (http://[::1]:12225/local/switch-id), retrying background_task = nat_v4_garbage_collector
138652025-04-30T06:43:17.696ZINFOtest_snapshot_basic (BackgroundTasks): Determining switch slots managed by switch zones background_task = bfd_manager
138662025-04-30T06:43:17.762ZINFOtest_snapshot_basic (BackgroundTasks): determining switch slot managed by dendrite zone background_task = bfd_manager zone_address = ::1
138672025-04-30T06:43:17.762ZDEBGtest_snapshot_basic (MgsClient): client request background_task = bfd_manager body = None method = GET uri = http://[::1]:12225/local/switch-id
138682025-04-30T06:43:17.762ZDEBGtest_snapshot_basic (MgsClient): client response background_task = bfd_manager result = Err(reqwest::Error { kind: Request, url: "http://[::1]:12225/local/switch-id", source: hyper_util::client::legacy::Error(Connect, ConnectError("tcp connect error", Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) })
138692025-04-30T06:43:17.762ZWARNtest_snapshot_basic (BackgroundTasks): failed to identify switch slot for dendrite background_task = bfd_manager reason = Communication Error: error sending request for url (http://[::1]:12225/local/switch-id): client error (Connect): tcp connect error: Connection refused (os error 146): Connection refused (os error 146) zone_address = ::1
138702025-04-30T06:43:17.767ZWARNtest_snapshot_basic (BackgroundTasks): Failed to map switch zone addr: Communication Error: error sending request for url (http://[::1]:12225/local/switch-id), retrying background_task = bfd_manager
138712025-04-30T06:43:17.767ZDEBGtest_snapshot_basic (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 6ad06c6b-9a6e-4ed1-b3f4-5d2e9da3f8a7 n_rows = 1 table_name = measurements_i64
138722025-04-30T06:43:17.767ZTRACtest_snapshot_basic (results-sink): successfully inserted samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
138732025-04-30T06:43:17.768ZINFOtest_snapshot_basic (BackgroundTasks): Determining switch slots managed by switch zones background_task = switch_port_config_manager rack_id = c19a698f-c6f9-4a17-ae30-20d711b8f7dc
138742025-04-30T06:43:17.825ZINFOtest_snapshot_basic (BackgroundTasks): determining switch slot managed by dendrite zone background_task = switch_port_config_manager rack_id = c19a698f-c6f9-4a17-ae30-20d711b8f7dc zone_address = ::1
138752025-04-30T06:43:17.825ZDEBGtest_snapshot_basic (MgsClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = c19a698f-c6f9-4a17-ae30-20d711b8f7dc uri = http://[::1]:12225/local/switch-id
138762025-04-30T06:43:17.825ZDEBGtest_snapshot_basic (MgsClient): client response background_task = switch_port_config_manager rack_id = c19a698f-c6f9-4a17-ae30-20d711b8f7dc result = Err(reqwest::Error { kind: Request, url: "http://[::1]:12225/local/switch-id", source: hyper_util::client::legacy::Error(Connect, ConnectError("tcp connect error", Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) })
138772025-04-30T06:43:17.826ZWARNtest_snapshot_basic (BackgroundTasks): failed to identify switch slot for dendrite background_task = switch_port_config_manager rack_id = c19a698f-c6f9-4a17-ae30-20d711b8f7dc reason = Communication Error: error sending request for url (http://[::1]:12225/local/switch-id): client error (Connect): tcp connect error: Connection refused (os error 146): Connection refused (os error 146) zone_address = ::1
138782025-04-30T06:43:17.828ZWARNtest_snapshot_basic (BackgroundTasks): Failed to map switch zone addr: Communication Error: error sending request for url (http://[::1]:12225/local/switch-id), retrying background_task = switch_port_config_manager rack_id = c19a698f-c6f9-4a17-ae30-20d711b8f7dc
138792025-04-30T06:43:17.835ZINFOtest_snapshot_basic (oximeter-agent): refreshing list of producers from Nexus collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 file = oximeter/collector/src/agent.rs:562
138802025-04-30T06:43:17.836ZDEBGtest_snapshot_basic (collection-task): sent timer-based collection request to the collection task collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
138812025-04-30T06:43:17.836ZWARNtest_snapshot_basic (results-sink-cluster): failed to insert some results into metric DB collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 error = Failed to check out connection to database: No backends found for this service file = oximeter/collector/src/results_sink.rs:92
138822025-04-30T06:43:17.836ZDEBGtest_snapshot_basic (collection-task): collecting from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
138832025-04-30T06:43:17.838ZDEBGtest_snapshot_basic (oximeter): client request body = None method = GET uri = http://127.0.0.1:61468/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
138842025-04-30T06:43:17.840ZTRACtest_snapshot_basic (dropshot_internal): incoming request local_addr = 127.0.0.1:61468 method = GET remote_addr = 127.0.0.1:64332 req_id = 5dfff9ef-62e6-43ac-953a-199dbb44aaf7 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
138852025-04-30T06:43:17.845ZTRACtest_snapshot_basic (dropshot_internal): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:61468 method = GET remote_addr = 127.0.0.1:64332 req_id = 5dfff9ef-62e6-43ac-953a-199dbb44aaf7 resource = Database uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
138862025-04-30T06:43:17.845ZDEBGtest_snapshot_basic (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:61468 method = GET remote_addr = 127.0.0.1:64332 req_id = 5dfff9ef-62e6-43ac-953a-199dbb44aaf7 roles = RoleSet { roles: {} } uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
138872025-04-30T06:43:17.847ZDEBGtest_snapshot_basic (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:61468 method = GET remote_addr = 127.0.0.1:64332 req_id = 5dfff9ef-62e6-43ac-953a-199dbb44aaf7 resource = Database result = Ok(()) uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
138882025-04-30T06:43:17.851ZTRACtest_snapshot_basic (collection-task): sent collection request to producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
138892025-04-30T06:43:17.851ZDEBGtest_snapshot_basic (collection-task): collected results from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 n_results = 1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
138902025-04-30T06:43:17.852ZTRACtest_snapshot_basic (collection-task): forwarded results to main collection loop collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
138912025-04-30T06:43:17.852ZTRACtest_snapshot_basic (collection-task): top of inner collection loop, waiting for next request collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
138922025-04-30T06:43:17.855ZINFOtest_snapshot_basic (dropshot_internal): request completed latency_us = 14929 local_addr = 127.0.0.1:61468 method = GET remote_addr = 127.0.0.1:64332 req_id = 5dfff9ef-62e6-43ac-953a-199dbb44aaf7 response_code = 200 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
138932025-04-30T06:43:17.856ZDEBGtest_snapshot_basic (oximeter): client response result = Ok(Response { url: "http://127.0.0.1:61468/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending", status: 200, headers: {"content-type": "application/json", "x-request-id": "5dfff9ef-62e6-43ac-953a-199dbb44aaf7", "content-length": "405", "date": "Wed, 30 Apr 2025 06:43:17 GMT"} })
138942025-04-30T06:43:17.857ZDEBGtest_snapshot_basic (oximeter): client request body = None method = GET uri = http://127.0.0.1:61468/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
138952025-04-30T06:43:17.858ZTRACtest_snapshot_basic (dropshot_internal): incoming request local_addr = 127.0.0.1:61468 method = GET remote_addr = 127.0.0.1:64332 req_id = 1a94b743-f1d4-4589-9274-d6f5aa48efeb uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
138962025-04-30T06:43:17.860ZTRACtest_snapshot_basic (dropshot_internal): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:61468 method = GET remote_addr = 127.0.0.1:64332 req_id = 1a94b743-f1d4-4589-9274-d6f5aa48efeb resource = Database uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
138972025-04-30T06:43:17.861ZDEBGtest_snapshot_basic (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:61468 method = GET remote_addr = 127.0.0.1:64332 req_id = 1a94b743-f1d4-4589-9274-d6f5aa48efeb roles = RoleSet { roles: {} } uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
138982025-04-30T06:43:17.861ZDEBGtest_snapshot_basic (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:61468 method = GET remote_addr = 127.0.0.1:64332 req_id = 1a94b743-f1d4-4589-9274-d6f5aa48efeb resource = Database result = Ok(()) uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
138992025-04-30T06:43:17.877ZINFOtest_snapshot_basic (dropshot_internal): request completed latency_us = 18726 local_addr = 127.0.0.1:61468 method = GET remote_addr = 127.0.0.1:64332 req_id = 1a94b743-f1d4-4589-9274-d6f5aa48efeb response_code = 200 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
139002025-04-30T06:43:17.878ZDEBGtest_snapshot_basic (oximeter): client response result = Ok(Response { url: "http://127.0.0.1:61468/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D", status: 200, headers: {"content-type": "application/json", "x-request-id": "1a94b743-f1d4-4589-9274-d6f5aa48efeb", "content-length": "29", "date": "Wed, 30 Apr 2025 06:43:17 GMT"} })
139012025-04-30T06:43:17.878ZTRACtest_snapshot_basic (oximeter-agent): ignoring request to update existing metric producer, since the endpoint information is the same as the existing collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = 354a52d4-3078-4706-81ce-12c32f6bbeaa
139022025-04-30T06:43:17.878ZTRACtest_snapshot_basic (oximeter-agent): ignoring request to update existing metric producer, since the endpoint information is the same as the existing collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
139032025-04-30T06:43:17.878ZINFOtest_snapshot_basic (oximeter-agent): refreshed list of producers from Nexus collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 file = oximeter/collector/src/agent.rs:637 n_current_tasks = 2 n_pruned_tasks = 0
139042025-04-30T06:43:17.885ZWARNtest_snapshot_basic (datastore): Non-retryable transaction failure err = DatabaseError(UniqueViolation, "duplicate key value violates unique constraint \\"inv_caboose_pkey\\"") transaction = inventory_insert_collection
139052025-04-30T06:43:17.886ZWARNtest_snapshot_basic (BackgroundTasks): inventory collection failed background_task = inventory_collection error = failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint "inv_caboose_pkey"
139062025-04-30T06:43:17.886ZDEBGtest_snapshot_basic (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 1.27490264s iteration = 50 status = {"error":"failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint \\"inv_caboose_pkey\\""}
139072025-04-30T06:43:17.886ZDEBGtest_snapshot_basic (BackgroundTasks): activating background_task = inventory_collection iteration = 51 reason = Signaled
139082025-04-30T06:43:17.886ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
139092025-04-30T06:43:17.886ZTRACtest_snapshot_basic (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
139102025-04-30T06:43:17.886ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
139112025-04-30T06:43:17.886ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
139122025-04-30T06:43:17.887ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
139132025-04-30T06:43:17.891ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
139142025-04-30T06:43:17.896ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
139152025-04-30T06:43:17.896ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
139162025-04-30T06:43:17.896ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
139172025-04-30T06:43:17.897ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
139182025-04-30T06:43:17.906ZDEBGtest_snapshot_basic (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = [(d1ad44e5-18dd-4193-a70e-bf9042f0e0e1, 0)]
139192025-04-30T06:43:17.906ZTRACtest_snapshot_basic (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
139202025-04-30T06:43:17.906ZTRACtest_snapshot_basic (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: 76 a1 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 v............_mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:63130 req_id = aa8352e2-6397-48dc-8e52-b1c226af7607
139212025-04-30T06:43:17.906ZDEBGtest_snapshot_basic (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 30369,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:63130 req_id = aa8352e2-6397-48dc-8e52-b1c226af7607
139222025-04-30T06:43:17.907ZDEBGtest_snapshot_basic (store): query key key = _mgs._tcp
139232025-04-30T06:43:17.907ZDEBGtest_snapshot_basic (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
139242025-04-30T06:43:17.907ZDEBGtest_snapshot_basic (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:63130 query = LowerQuery { name: LowerName(Name("_mgs._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 41502, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = aa8352e2-6397-48dc-8e52-b1c226af7607
139252025-04-30T06:43:17.907ZDEBGtest_snapshot_basic (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 41502, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 4262, tv_nsec: 706627926 } })
139262025-04-30T06:43:17.907ZTRACtest_snapshot_basic (DnsResolver): lookup_service_targets: looking up SRV target
139272025-04-30T06:43:17.908ZDEBGtest_snapshot_basic: attempt to fetch latest inventory collection result = Ok("found")
139282025-04-30T06:43:17.908ZTRACtest_snapshot_basic (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: 3f e5 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e ?...........-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:51170 req_id = 6b7f3952-6026-4748-9381-153f5ad9597c
139292025-04-30T06:43:17.908ZDEBGtest_snapshot_basic (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 16357,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:51170 req_id = 6b7f3952-6026-4748-9381-153f5ad9597c
139302025-04-30T06:43:17.908ZDEBGtest_snapshot_basic (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
139312025-04-30T06:43:17.908ZDEBGtest_snapshot_basic (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:51170 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = 6b7f3952-6026-4748-9381-153f5ad9597c
139322025-04-30T06:43:17.960ZTRACtest_snapshot_basic (DnsResolver): lookup_all_socket_v6 srv dns_name = _clickhouse-admin-keeper._tcp.control-plane.oxide.internal
139332025-04-30T06:43:17.961ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
139342025-04-30T06:43:17.961ZTRACtest_snapshot_basic (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
139352025-04-30T06:43:17.961ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
139362025-04-30T06:43:17.961ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {} }
139372025-04-30T06:43:17.963ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
139382025-04-30T06:43:17.963ZTRACtest_snapshot_basic (dns): buffer buffer = Length: 87 (0x57) bytes\n0000: 29 c2 01 00 00 01 00 00 00 00 00 01 18 5f 63 6c )............_cl\n0010: 69 63 6b 68 6f 75 73 65 2d 61 64 6d 69 6e 2d 6b ickhouse-admin-k\n0020: 65 65 70 65 72 04 5f 74 63 70 0d 63 6f 6e 74 72 eeper._tcp.contr\n0030: 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 69 ol-plane.oxide.i\n0040: 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 04 nternal..!....).\n0050: d0 00 00 00 00 00 00 ....... kind = dns peer_addr = [::1]:53693 req_id = 942a60ff-37b1-40ff-9bcc-9568bce1b59d
139392025-04-30T06:43:17.963ZDEBGtest_snapshot_basic (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 10690,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 24,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 97,\n 100,\n 109,\n 105,\n 110,\n 45,\n 107,\n 101,\n 101,\n 112,\n 101,\n 114,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:53693 req_id = 942a60ff-37b1-40ff-9bcc-9568bce1b59d
139402025-04-30T06:43:17.964ZDEBGtest_snapshot_basic (store): query key key = _clickhouse-admin-keeper._tcp
139412025-04-30T06:43:17.964ZERROtest_snapshot_basic (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 10690, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } NXDOMAIN: no records found for name: "_clickhouse-admin-keeper._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:53693 req_id = 942a60ff-37b1-40ff-9bcc-9568bce1b59d
139422025-04-30T06:43:17.965ZDEBGtest_snapshot_basic (BackgroundTasks): begin collection background_task = inventory_collection
139432025-04-30T06:43:17.965ZDEBGtest_snapshot_basic (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:41502
139442025-04-30T06:43:17.965ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/ignition
139452025-04-30T06:43:17.966ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
139462025-04-30T06:43:17.974ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
139472025-04-30T06:43:17.974ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
139482025-04-30T06:43:17.974ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {} }
139492025-04-30T06:43:17.975ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
139502025-04-30T06:43:17.977ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "7177d4fe-65a7-4f20-b56c-c19b598d6c54", "content-length": "835", "date": "Wed, 30 Apr 2025 06:43:17 GMT"} })
139512025-04-30T06:43:17.977ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/0
139522025-04-30T06:43:17.978ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
139532025-04-30T06:43:17.978ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {} }
139542025-04-30T06:43:17.978ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
139552025-04-30T06:43:17.981ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "ad1b0d15-7d82-4cad-951f-933cded17015", "content-length": "734", "date": "Wed, 30 Apr 2025 06:43:17 GMT"} })
139562025-04-30T06:43:17.981ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/0/component/sp/caboose?firmware_slot=0
139572025-04-30T06:43:17.985ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "f93f31d8-e80f-4b8b-a733-dce55d96169a", "content-length": "111", "date": "Wed, 30 Apr 2025 06:43:17 GMT"} })
139582025-04-30T06:43:17.985ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/0/component/sp/caboose?firmware_slot=1
139592025-04-30T06:43:17.990ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "43fb8071-4de3-484a-83ce-5d94cdebb81f", "content-length": "111", "date": "Wed, 30 Apr 2025 06:43:17 GMT"} })
139602025-04-30T06:43:17.990ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/0/component/rot/caboose?firmware_slot=0
139612025-04-30T06:43:17.994ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "94e16271-8968-4c9c-a2e3-130447a0671d", "content-length": "170", "date": "Wed, 30 Apr 2025 06:43:17 GMT"} })
139622025-04-30T06:43:17.994ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/0/component/rot/caboose?firmware_slot=1
139632025-04-30T06:43:17.997ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "d2bc1195-77e8-4075-b8f1-e9e3a253b413", "content-length": "170", "date": "Wed, 30 Apr 2025 06:43:17 GMT"} })
139642025-04-30T06:43:17.997ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/0/component/stage0/caboose?firmware_slot=0
139652025-04-30T06:43:18.002ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "98dcc0a1-42f5-4ad7-8573-79d6ac559e4c", "content-length": "179", "date": "Wed, 30 Apr 2025 06:43:17 GMT"} })
139662025-04-30T06:43:18.002ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/0/component/stage0/caboose?firmware_slot=1
139672025-04-30T06:43:18.004ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "3c8ba2cd-2bad-4bde-a7ae-7a7d75c6979c", "content-length": "179", "date": "Wed, 30 Apr 2025 06:43:17 GMT"} })
139682025-04-30T06:43:18.004ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/0/component/rot/cmpa
139692025-04-30T06:43:18.006ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "f5493d47-0b0f-4ac2-978a-48244d1e7a44", "content-length": "702", "date": "Wed, 30 Apr 2025 06:43:17 GMT"} })
139702025-04-30T06:43:18.006ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/0/component/rot/cfpa
139712025-04-30T06:43:18.008ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "3be00e93-1380-4484-8e64-07f8a86b44f2", "content-length": "727", "date": "Wed, 30 Apr 2025 06:43:17 GMT"} })
139722025-04-30T06:43:18.008ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/0/component/rot/cfpa
139732025-04-30T06:43:18.010ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "f0f29eef-b977-49ed-bf21-2d64303dcb81", "content-length": "729", "date": "Wed, 30 Apr 2025 06:43:17 GMT"} })
139742025-04-30T06:43:18.011ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/0/component/rot/cfpa
139752025-04-30T06:43:18.018ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "1c0d4381-64bd-4aae-9db0-60a6f193cccb", "content-length": "728", "date": "Wed, 30 Apr 2025 06:43:17 GMT"} })
139762025-04-30T06:43:18.019ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/1
139772025-04-30T06:43:18.020ZDEBGtest_snapshot_basic: attempt to fetch latest inventory collection result = Ok("found")
139782025-04-30T06:43:18.023ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "9cbddffb-6b1d-4e74-9817-78bda48e9561", "content-length": "734", "date": "Wed, 30 Apr 2025 06:43:17 GMT"} })
139792025-04-30T06:43:18.023ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/1/component/sp/caboose?firmware_slot=0
139802025-04-30T06:43:18.025ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "89fcc9d1-afbe-46b9-8b62-db52833a9df6", "content-length": "111", "date": "Wed, 30 Apr 2025 06:43:17 GMT"} })
139812025-04-30T06:43:18.026ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/1/component/sp/caboose?firmware_slot=1
139822025-04-30T06:43:18.031ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "9851bf60-027c-4a2f-a4f4-ecf395496f17", "content-length": "111", "date": "Wed, 30 Apr 2025 06:43:17 GMT"} })
139832025-04-30T06:43:18.031ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/1/component/rot/caboose?firmware_slot=0
139842025-04-30T06:43:18.034ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "7ed8cc6a-7cd1-495c-9bcf-065e59817a86", "content-length": "170", "date": "Wed, 30 Apr 2025 06:43:17 GMT"} })
139852025-04-30T06:43:18.034ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/1/component/rot/caboose?firmware_slot=1
139862025-04-30T06:43:18.035ZDEBGtest_snapshot_basic (results-sink): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
139872025-04-30T06:43:18.035ZDEBGtest_snapshot_basic (results-sink-cluster): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
139882025-04-30T06:43:18.035ZDEBGtest_snapshot_basic (clickhouse-client): unrolling 1 total samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 6ad06c6b-9a6e-4ed1-b3f4-5d2e9da3f8a7
139892025-04-30T06:43:18.037ZTRACtest_snapshot_basic (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 6ad06c6b-9a6e-4ed1-b3f4-5d2e9da3f8a7 n_columns = 4 n_rows = 2 sql = INSERT INTO oximeter.fields_string FORMAT Native
139902025-04-30T06:43:18.039ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "9a6c524b-6e25-4acb-bdc0-fdb659b8edd4", "content-length": "170", "date": "Wed, 30 Apr 2025 06:43:17 GMT"} })
139912025-04-30T06:43:18.039ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/1/component/stage0/caboose?firmware_slot=0
139922025-04-30T06:43:18.041ZDEBGtest_snapshot_basic (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 6ad06c6b-9a6e-4ed1-b3f4-5d2e9da3f8a7 n_rows = 2 table_name = fields_string
139932025-04-30T06:43:18.041ZTRACtest_snapshot_basic (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 6ad06c6b-9a6e-4ed1-b3f4-5d2e9da3f8a7 n_columns = 4 n_rows = 1 sql = INSERT INTO oximeter.measurements_i64 FORMAT Native
139942025-04-30T06:43:18.045ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "552b7782-34ad-4c8c-aedc-e422dffafaf6", "content-length": "179", "date": "Wed, 30 Apr 2025 06:43:17 GMT"} })
139952025-04-30T06:43:18.045ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/1/component/stage0/caboose?firmware_slot=1
139962025-04-30T06:43:18.048ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "7af78dd6-de42-4b52-98f1-a1d6fca22b51", "content-length": "179", "date": "Wed, 30 Apr 2025 06:43:17 GMT"} })
139972025-04-30T06:43:18.048ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/1/component/rot/cmpa
139982025-04-30T06:43:18.049ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "87d4cf13-28e0-4f30-ba38-0626226ded6f", "content-length": "702", "date": "Wed, 30 Apr 2025 06:43:17 GMT"} })
139992025-04-30T06:43:18.049ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/1/component/rot/cfpa
140002025-04-30T06:43:18.050ZDEBGtest_snapshot_basic (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 6ad06c6b-9a6e-4ed1-b3f4-5d2e9da3f8a7 n_rows = 1 table_name = measurements_i64
140012025-04-30T06:43:18.050ZTRACtest_snapshot_basic (results-sink): successfully inserted samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
140022025-04-30T06:43:18.051ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "8737e4f0-865d-4c3b-b50c-bd82a10fc1e9", "content-length": "727", "date": "Wed, 30 Apr 2025 06:43:17 GMT"} })
140032025-04-30T06:43:18.051ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/1/component/rot/cfpa
140042025-04-30T06:43:18.052ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "96b829db-f9ce-4802-93f2-6dfed16ec326", "content-length": "729", "date": "Wed, 30 Apr 2025 06:43:17 GMT"} })
140052025-04-30T06:43:18.052ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/switch/1/component/rot/cfpa
140062025-04-30T06:43:18.053ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "838b5a2c-52ab-42b8-b13e-a7d24c3378b8", "content-length": "728", "date": "Wed, 30 Apr 2025 06:43:17 GMT"} })
140072025-04-30T06:43:18.054ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/0
140082025-04-30T06:43:18.056ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "f0b74a3a-ee4e-4a90-b2cd-137ebe0fce4f", "content-length": "723", "date": "Wed, 30 Apr 2025 06:43:17 GMT"} })
140092025-04-30T06:43:18.056ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/0/component/sp/caboose?firmware_slot=0
140102025-04-30T06:43:18.059ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "3ef01e3b-a226-44e5-b4c1-09e57205535e", "content-length": "109", "date": "Wed, 30 Apr 2025 06:43:17 GMT"} })
140112025-04-30T06:43:18.059ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/0/component/sp/caboose?firmware_slot=1
140122025-04-30T06:43:18.062ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "62844b1f-e994-4cd6-8098-b981b1b99a61", "content-length": "109", "date": "Wed, 30 Apr 2025 06:43:17 GMT"} })
140132025-04-30T06:43:18.062ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/0/component/rot/caboose?firmware_slot=0
140142025-04-30T06:43:18.064ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "3e98be4f-8c04-4711-aea8-a16d7e7a581c", "content-length": "169", "date": "Wed, 30 Apr 2025 06:43:17 GMT"} })
140152025-04-30T06:43:18.064ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/0/component/rot/caboose?firmware_slot=1
140162025-04-30T06:43:18.066ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "c0bbc563-e5c4-4c05-9e67-9ef499efd191", "content-length": "169", "date": "Wed, 30 Apr 2025 06:43:17 GMT"} })
140172025-04-30T06:43:18.066ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/0/component/stage0/caboose?firmware_slot=0
140182025-04-30T06:43:18.068ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "65f0272e-af69-4038-b23d-590e0b51dc3c", "content-length": "178", "date": "Wed, 30 Apr 2025 06:43:17 GMT"} })
140192025-04-30T06:43:18.068ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/0/component/stage0/caboose?firmware_slot=1
140202025-04-30T06:43:18.070ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "8383e178-96b2-48d3-8ea0-f3420d86ab18", "content-length": "178", "date": "Wed, 30 Apr 2025 06:43:17 GMT"} })
140212025-04-30T06:43:18.070ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/0/component/rot/cmpa
140222025-04-30T06:43:18.071ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
140232025-04-30T06:43:18.071ZTRACtest_snapshot_basic (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
140242025-04-30T06:43:18.071ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
140252025-04-30T06:43:18.071ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {} }
140262025-04-30T06:43:18.072ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
140272025-04-30T06:43:18.073ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "821fe43a-7f2f-4161-a008-9181cabcc952", "content-length": "702", "date": "Wed, 30 Apr 2025 06:43:17 GMT"} })
140282025-04-30T06:43:18.073ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/0/component/rot/cfpa
140292025-04-30T06:43:18.077ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "3b75990d-1845-49d1-8eb6-7a84b2fb4d8b", "content-length": "727", "date": "Wed, 30 Apr 2025 06:43:17 GMT"} })
140302025-04-30T06:43:18.077ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/0/component/rot/cfpa
140312025-04-30T06:43:18.077ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
140322025-04-30T06:43:18.087ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
140332025-04-30T06:43:18.087ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
140342025-04-30T06:43:18.087ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {} }
140352025-04-30T06:43:18.088ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
140362025-04-30T06:43:18.090ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "a359f5ea-84b0-4b6d-8ce9-c7f4fbfb2a85", "content-length": "729", "date": "Wed, 30 Apr 2025 06:43:17 GMT"} })
140372025-04-30T06:43:18.091ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/0/component/rot/cfpa
140382025-04-30T06:43:18.092ZTRACtest_snapshot_basic (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
140392025-04-30T06:43:18.092ZDEBGtest_snapshot_basic (InternalApi): roles roles = RoleSet { roles: {} }
140402025-04-30T06:43:18.092ZDEBGtest_snapshot_basic (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
140412025-04-30T06:43:18.093ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "3ab46384-dc6e-4c5a-9f6e-0d24b1120393", "content-length": "728", "date": "Wed, 30 Apr 2025 06:43:17 GMT"} })
140422025-04-30T06:43:18.094ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/1
140432025-04-30T06:43:18.095ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "8004afc8-874e-4c8f-9fe4-bf92e9b71bc6", "content-length": "723", "date": "Wed, 30 Apr 2025 06:43:17 GMT"} })
140442025-04-30T06:43:18.095ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/1/component/sp/caboose?firmware_slot=0
140452025-04-30T06:43:18.102ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "2e084be0-75cf-4fff-93b8-6abcca3b17be", "content-length": "109", "date": "Wed, 30 Apr 2025 06:43:17 GMT"} })
140462025-04-30T06:43:18.103ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/1/component/sp/caboose?firmware_slot=1
140472025-04-30T06:43:18.108ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "02052365-1c0c-437f-991a-a55a599c0f75", "content-length": "109", "date": "Wed, 30 Apr 2025 06:43:17 GMT"} })
140482025-04-30T06:43:18.108ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/1/component/rot/caboose?firmware_slot=0
140492025-04-30T06:43:18.111ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "965994ff-79f0-4d49-80e6-35444983f142", "content-length": "169", "date": "Wed, 30 Apr 2025 06:43:17 GMT"} })
140502025-04-30T06:43:18.112ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/1/component/rot/caboose?firmware_slot=1
140512025-04-30T06:43:18.116ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "43a2b795-e96a-4003-9b96-177b23f6d8c3", "content-length": "169", "date": "Wed, 30 Apr 2025 06:43:17 GMT"} })
140522025-04-30T06:43:18.117ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/1/component/stage0/caboose?firmware_slot=0
140532025-04-30T06:43:18.118ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "4fcf7c31-d1b3-4e7e-9194-3cda34b96832", "content-length": "178", "date": "Wed, 30 Apr 2025 06:43:17 GMT"} })
140542025-04-30T06:43:18.118ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/1/component/stage0/caboose?firmware_slot=1
140552025-04-30T06:43:18.121ZDEBGtest_snapshot_basic: attempt to fetch latest inventory collection result = Ok("found")
140562025-04-30T06:43:18.123ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "6d59f3b1-7579-44c1-84a1-90f8cdff24df", "content-length": "178", "date": "Wed, 30 Apr 2025 06:43:17 GMT"} })
140572025-04-30T06:43:18.123ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/1/component/rot/cmpa
140582025-04-30T06:43:18.124ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "0df3d1b7-ec0e-4716-902a-cc44059fb939", "content-length": "702", "date": "Wed, 30 Apr 2025 06:43:17 GMT"} })
140592025-04-30T06:43:18.124ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/1/component/rot/cfpa
140602025-04-30T06:43:18.125ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "1262b808-53b7-481f-b079-6814d1caacc9", "content-length": "727", "date": "Wed, 30 Apr 2025 06:43:17 GMT"} })
140612025-04-30T06:43:18.125ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/1/component/rot/cfpa
140622025-04-30T06:43:18.126ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "c9ead5ff-0602-4a83-abd5-0b9769eca6c0", "content-length": "729", "date": "Wed, 30 Apr 2025 06:43:17 GMT"} })
140632025-04-30T06:43:18.126ZDEBGtest_snapshot_basic (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41502 method = GET uri = http://[::1]:41502/sp/sled/1/component/rot/cfpa
140642025-04-30T06:43:18.127ZDEBGtest_snapshot_basic (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41502 result = Ok(Response { url: "http://[::1]:41502/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "9429fee5-276f-4c6e-be7a-9dfcb290867e", "content-length": "728", "date": "Wed, 30 Apr 2025 06:43:17 GMT"} })
140652025-04-30T06:43:18.127ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
140662025-04-30T06:43:18.128ZTRACtest_snapshot_basic (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
140672025-04-30T06:43:18.128ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
140682025-04-30T06:43:18.128ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
140692025-04-30T06:43:18.129ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
140702025-04-30T06:43:18.133ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
140712025-04-30T06:43:18.137ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
140722025-04-30T06:43:18.137ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
140732025-04-30T06:43:18.137ZTRACtest_snapshot_basic (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
140742025-04-30T06:43:18.137ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
140752025-04-30T06:43:18.137ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
140762025-04-30T06:43:18.139ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
140772025-04-30T06:43:18.140ZWARNtest_snapshot_basic (results-sink-cluster): failed to insert some results into metric DB collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 error = Failed to check out connection to database: No backends found for this service file = oximeter/collector/src/results_sink.rs:92
140782025-04-30T06:43:18.142ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
140792025-04-30T06:43:18.147ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
140802025-04-30T06:43:18.147ZTRACtest_snapshot_basic (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
140812025-04-30T06:43:18.147ZDEBGtest_snapshot_basic (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
140822025-04-30T06:43:18.148ZDEBGtest_snapshot_basic (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
140832025-04-30T06:43:18.237ZDEBGtest_snapshot_basic (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:41923
140842025-04-30T06:43:18.237ZDEBGtest_snapshot_basic (BackgroundTasks): client request SledAgent = http://[::1]:41923 background_task = inventory_collection body = None method = GET uri = http://[::1]:41923/inventory